acc vs acc_norm: Why Length Bias Skews LLM Eval Scores

A developer explains how the choice between `acc` and `acc_norm` in lm-eval-harness can skew LLM evaluation results due to length bias. The raw `acc` metric favors shorter answers because it sums token log-probabilities, which grow with length, while `acc_norm` divides by byte length to reduce this bias. The post demonstrates that different scoring rules can rank candidates differently and recommends reporting both metrics and cleaning choice strings.

Your fine-tune gains three points of acc norm on HellaSwag and loses two points of acc . Same checkpoint, same harness, same seed. Nothing about the model's commonsense reasoning moved in two directions at once — you changed its average per-token entropy, and one of those two metrics is partly measuring how long the answer string is. This is the acc vs acc norm problem, and it decides which number goes in your model card. lm-eval-harness multiple-choice tasks, acc picks the candidate with the highest acc norm divides that sum by the - length x average per-token entropy , so it structurally prefers acc is measuring length as much as knowledge. acc and acc norm produce identical rankings. acc and acc norm in opposite directions with no capability change. Report both, and pick one metric per task family before you start training.Both are argmax over candidate continuations. Given a context x and candidates y 1 ... y k , the harness runs one forward pass per candidate and computes the summed token log-probability of the continuation: score raw y = sum t log p y t | x, y <t acc is argmax y score raw y . acc norm is argmax y score raw y / len y.encode "utf-8" . That is the whole difference: a denominator. It matters enormously because the numerator is an extensive quantity — it grows with length — while the thing you want to measure is not. Because every additional token adds another negative term. For a token sequence of length L , the expected summed log-probability under the model's own distribution is -L x H , where H is the average per-token entropy the model assigns along that path. Well-calibrated English text under an 8B model sits somewhere in the low single digits of nats per token. Adding ten tokens to a candidate costs on the order of ten to twenty nats — far more than the few nats of signal that distinguish a plausible ending from an implausible one. So on any task where the correct answer is not systematically the shortest, acc throws away accuracy for free. Here is the shape of it, with illustrative numbers not measured — the point is the ordering, not the magnitudes : | candidate | tokens | bytes | sum logprob | per-token | per-byte | |---|---|---|---|---|---| | A: " He leaves." wrong | 4 | 11 | -8.0 | -2.00 | -0.727 | | B: " He picks up the towel and folds it neatly." correct | 11 | 42 | -14.5 | -1.32 | -0.345 | | C: " He defenestrates it." wrong, rare word | 9 | 21 | -11.0 | -1.22 | -0.524 | Raw sum picks A. Per-byte picks B. Per-token picks C. Three defensible-looking scoring rules, three different answers, one forward pass each. Because token counts are a property of the tokenizer, not the text. Candidate C above is the failure mode: a rare word explodes into many subword pieces, and each of those pieces is highly predictable given the previous piece . defenestr - ates is nearly free. So the sum is spread over a large denominator of cheap tokens, and per-token normalization hands a rare, wrong answer an artificial win. Byte length is tokenizer-independent. That is the real argument for it: it lets you put a Llama-tokenized model and a Gemma-tokenized model in the same table without the fertility difference leaking into the metric. It is not a principled information-theoretic correction — bits-per-byte is a proper cross-tokenizer measure, but acc norm is applying it as a heuristic tiebreaker over candidates, not as a likelihood. Practical consequence: the denominator is computed on the raw string as it appears in the dataset . Trailing whitespace, a period the annotator forgot, a leading space that your prompt template does or does not consume — each of those shifts the denominator by a byte or two. For a five-character candidate, one byte is a 20% change in the normalized score. Clean your choice strings. Three scorings from one forward pass: python import torch, torch.nn.functional as F from transformers import AutoModelForCausalLM, AutoTokenizer model id = "meta-llama/Llama-3.1-8B" tok = AutoTokenizer.from pretrained model id model = AutoModelForCausalLM.from pretrained model id, torch dtype=torch.bfloat16, device map="cuda" .eval @torch.no grad def score context: str, continuation: str - dict: ctx ids = tok context, add special tokens=True .input ids full ids = tok context + continuation, add special tokens=True .input ids n cont = len full ids - len ctx ids inp = torch.tensor full ids , device=model.device logits = model inp .logits 0 .float logprobs = F.log softmax logits :-1 , dim=-1 position i predicts token i+1 tok ll = logprobs.gather -1, inp 0, 1: .unsqueeze -1 .squeeze -1 cont ll = tok ll len ctx ids - 1: logprobs of continuation tokens total = cont ll.sum .item return { "acc": total, raw sum "per token": total / n cont, "acc norm": total / len continuation.encode "utf-8" , byte-normalized } def predict context, choices, metric="acc norm" : return max range len choices , key=lambda i: score context, choices i metric One caveat that bites people writing their own harness: splitting on len ctx ids assumes tokenizing context + continuation reproduces the context tokens exactly. It usually does when the continuation starts with a space and the context ends with a non-space, and it silently does not when the boundary falls inside a merge. The harness handles this by encoding the pair and aligning; if you roll your own, assert that full ids :len ctx ids == ctx ids and fail loudly. Because MMLU as scored in the harness uses single-letter continuations — doc to choice: "A", "B", "C", "D" . Every candidate is one character, so every denominator is 1, so normalization is a no-op and the argmax is identical. The two reported numbers are the same number. This is why "MMLU is length-robust" and "HellaSwag needs acc norm " are both true and not in tension. The difference is the answer format , not the benchmark's difficulty. Any task you author with letter-key answers is immune; any task that scores full natural-language continuations is exposed. It also means you cannot compare a letter-key MMLU run against a full-text-continuation MMLU variant. They are different measurements sharing a name. Because SFT and preference optimization change per-token entropy, and the length penalty in raw acc scales with entropy. An instruction-tuned model is sharper on fluent, in-distribution continuations than the base model was. Sharper means each token costs fewer nats, which means the per-token penalty on long candidates shrinks, which means raw acc mechanically drifts up on tasks with long correct answers — with zero change in what the model knows. The same effect runs backwards when your SFT data pushes the model toward a format the eval candidates do not match. Now the candidates are out-of-distribution, entropy rises, long answers get hammered, acc collapses, and acc norm barely moves because the denominator absorbs part of it. Rule of thumb: if acc and acc norm move in the same direction, you probably changed capability. If they diverge, you probably changed the output distribution's sharpness or format. Check with a small run on a letter-key version of the same questions — if that number is flat, it was never capability. When candidate answers differ in raw frequency more than in length. Byte normalization does nothing about a candidate that is simply a more common string in pretraining data. The fix from the GPT-3 era is unconditional pointwise-mutual-information style normalization: score the continuation against a null context and subtract. score pmi y = log p y | x - log p y | "Answer:" This costs a second forward pass per candidate — often cacheable, since the null context is shared — and it targets a genuinely different bias. Use it on tasks whose distractors are high-frequency phrases many ARC-Easy items and where length variance is small. Do not stack it blindly on top of byte normalization; you will double-correct and the ranking becomes hard to reason about. For a custom multiple-choice task, output type: multiple choice computes both metrics; you choose which to list: task: my mc task dataset path: json dataset kwargs: data files: test: data/mc.jsonl test split: test output type: multiple choice doc to text: "Question: {{question}}\nAnswer:" doc to choice: "{{choices}}" e.g. " Paris", " Lyon", " Marseille" doc to target: "{{label}}" integer index metric list: - metric: acc aggregation: mean higher is better: true - metric: acc norm aggregation: mean higher is better: true Two details worth enforcing in review. First, doc to text ends with Answer: and every choice starts with a space — keep the space on the choice side, consistently, or your denominators and your tokenization both wobble per-item. Second, if your choices are letter keys, drop acc norm from the list entirely; reporting a metric that is provably identical to another one just invites someone to quote whichever is higher. A third, unrelated meaning of "normalized" shows up at the leaderboard level: rescaling a task's score against its random baseline so a 4-way multiple-choice task does not start at 25%. That happens after aggregation and has nothing to do with acc norm . Do not conflate them in a results table. acc and acc norm disagree because summed log-likelihood is an extensive quantity that grows with continuation length, so raw acc systematically favors short candidates, while acc norm divides by the continuation's byte length to remove most of that length bias. Use acc norm for any task scoring full natural-language continuations of varying length HellaSwag, ARC, PIQA , where it is the more honest measure of what the model knows; use plain acc for letter-key formats like MMLU, where the two are numerically identical anyway; reach for PMI-style unconditional normalization when the confound is candidate frequency rather than candidate length. Fix your choice per task family before training starts, report both numbers, and treat a divergence between them as a signal that your model's output distribution changed shape — not that it got smarter.