# acc vs acc_norm: Why Length Bias Skews LLM Eval Scores > Source: > Published: 2026-08-20 15:20:49+00:00 Your fine-tune gains three points of `acc_norm` on HellaSwag and loses two points of `acc` . Same checkpoint, same harness, same seed. Nothing about the model's commonsense reasoning moved in two directions at once — you changed its average per-token entropy, and one of those two metrics is partly measuring how long the answer string is. This is the `acc vs acc_norm` problem, and it decides which number goes in your model card. `lm-eval-harness` multiple-choice tasks, `acc` picks the candidate with the highest `acc_norm` divides that sum by the `-(length) x (average per-token entropy)` , so it structurally prefers `acc` is measuring length as much as knowledge.`acc` and `acc_norm` produce identical rankings.`acc` and `acc_norm` in opposite directions with no capability change. Report both, and pick one metric per task family before you start training.Both are argmax over candidate continuations. Given a context `x` and candidates `y_1 ... y_k` , the harness runs one forward pass per candidate and computes the summed token log-probability of the continuation: ``` score_raw(y) = sum_t log p(y_t | x, y_ `ates` is nearly free. So the sum is spread over a large denominator of cheap tokens, and per-token normalization hands a rare, wrong answer an artificial win. Byte length is tokenizer-independent. That is the real argument for it: it lets you put a Llama-tokenized model and a Gemma-tokenized model in the same table without the fertility difference leaking into the metric. It is not a principled information-theoretic correction — bits-per-byte is a proper cross-tokenizer measure, but `acc_norm` is applying it as a heuristic tiebreaker over candidates, not as a likelihood. Practical consequence: the denominator is computed on the **raw string as it appears in the dataset**. Trailing whitespace, a period the annotator forgot, a leading space that your prompt template does or does not consume — each of those shifts the denominator by a byte or two. For a five-character candidate, one byte is a 20% change in the normalized score. Clean your choice strings. Three scorings from one forward pass: ``` python import torch, torch.nn.functional as F from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "meta-llama/Llama-3.1-8B" tok = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="cuda" ).eval() @torch.no_grad() def score(context: str, continuation: str) -> dict: ctx_ids = tok(context, add_special_tokens=True).input_ids full_ids = tok(context + continuation, add_special_tokens=True).input_ids n_cont = len(full_ids) - len(ctx_ids) inp = torch.tensor([full_ids], device=model.device) logits = model(inp).logits[0].float() logprobs = F.log_softmax(logits[:-1], dim=-1) # position i predicts token i+1 tok_ll = logprobs.gather(-1, inp[0, 1:].unsqueeze(-1)).squeeze(-1) cont_ll = tok_ll[len(ctx_ids) - 1:] # logprobs of continuation tokens total = cont_ll.sum().item() return { "acc": total, # raw sum "per_token": total / n_cont, "acc_norm": total / len(continuation.encode("utf-8")), # byte-normalized } def predict(context, choices, metric="acc_norm"): return max(range(len(choices)), key=lambda i: score(context, choices[i])[metric]) ``` One caveat that bites people writing their own harness: splitting on `len(ctx_ids)` assumes tokenizing `context + continuation` reproduces the context tokens exactly. It usually does when the continuation starts with a space and the context ends with a non-space, and it silently does not when the boundary falls inside a merge. The harness handles this by encoding the pair and aligning; if you roll your own, assert that `full_ids[:len(ctx_ids)] == ctx_ids` and fail loudly. Because MMLU as scored in the harness uses single-letter continuations — `doc_to_choice: ["A", "B", "C", "D"]` . Every candidate is one character, so every denominator is 1, so normalization is a no-op and the argmax is identical. The two reported numbers are the same number. This is why "MMLU is length-robust" and "HellaSwag needs `acc_norm` " are both true and not in tension. The difference is the answer *format*, not the benchmark's difficulty. Any task you author with letter-key answers is immune; any task that scores full natural-language continuations is exposed. It also means you cannot compare a letter-key MMLU run against a full-text-continuation MMLU variant. They are different measurements sharing a name. Because SFT and preference optimization change per-token entropy, and the length penalty in raw `acc` scales with entropy. An instruction-tuned model is sharper on fluent, in-distribution continuations than the base model was. Sharper means each token costs fewer nats, which means the per-token penalty on long candidates shrinks, which means raw `acc` mechanically drifts up on tasks with long correct answers — with zero change in what the model knows. The same effect runs backwards when your SFT data pushes the model toward a format the eval candidates do not match. Now the candidates are out-of-distribution, entropy rises, long answers get hammered, `acc` collapses, and `acc_norm` barely moves because the denominator absorbs part of it. Rule of thumb: if `acc` and `acc_norm` move in the same direction, you probably changed capability. If they diverge, you probably changed the output distribution's sharpness or format. Check with a small run on a letter-key version of the same questions — if that number is flat, it was never capability. When candidate answers differ in raw frequency more than in length. Byte normalization does nothing about a candidate that is simply a more common string in pretraining data. The fix from the GPT-3 era is unconditional (pointwise-mutual-information style) normalization: score the continuation against a null context and subtract. ``` score_pmi(y) = log p(y | x) - log p(y | "Answer:") ``` This costs a second forward pass per candidate — often cacheable, since the null context is shared — and it targets a genuinely different bias. Use it on tasks whose distractors are high-frequency phrases (many ARC-Easy items) and where length variance is small. Do not stack it blindly on top of byte normalization; you will double-correct and the ranking becomes hard to reason about. For a custom multiple-choice task, `output_type: multiple_choice` computes both metrics; you choose which to list: ``` task: my_mc_task dataset_path: json dataset_kwargs: data_files: test: data/mc.jsonl test_split: test output_type: multiple_choice doc_to_text: "Question: {{question}}\nAnswer:" doc_to_choice: "{{choices}}" # e.g. [" Paris", " Lyon", " Marseille"] doc_to_target: "{{label}}" # integer index metric_list: - metric: acc aggregation: mean higher_is_better: true - metric: acc_norm aggregation: mean higher_is_better: true ``` Two details worth enforcing in review. First, `doc_to_text` ends with `Answer:` and every choice starts with a space — keep the space on the choice side, consistently, or your denominators and your tokenization both wobble per-item. Second, if your choices are letter keys, drop `acc_norm` from the list entirely; reporting a metric that is provably identical to another one just invites someone to quote whichever is higher. A third, unrelated meaning of "normalized" shows up at the leaderboard level: rescaling a task's score against its random baseline so a 4-way multiple-choice task does not start at 25%. That happens after aggregation and has nothing to do with `acc_norm` . Do not conflate them in a results table. `acc` and `acc_norm` disagree because summed log-likelihood is an extensive quantity that grows with continuation length, so raw `acc` systematically favors short candidates, while `acc_norm` divides by the continuation's byte length to remove most of that length bias. Use `acc_norm` for any task scoring full natural-language continuations of varying length (HellaSwag, ARC, PIQA), where it is the more honest measure of what the model knows; use plain `acc` for letter-key formats like MMLU, where the two are numerically identical anyway; reach for PMI-style unconditional normalization when the confound is candidate frequency rather than candidate length. Fix your choice per task family before training starts, report both numbers, and treat a divergence between them as a signal that your model's output distribution changed shape — not that it got smarter.