{"slug": "acc-vs-acc-norm-why-length-bias-skews-llm-eval-scores", "title": "acc vs acc_norm: Why Length Bias Skews LLM Eval Scores", "summary": "A developer explains how the choice between `acc` and `acc_norm` in lm-eval-harness can skew LLM evaluation results due to length bias. The raw `acc` metric favors shorter answers because it sums token log-probabilities, which grow with length, while `acc_norm` divides by byte length to reduce this bias. The post demonstrates that different scoring rules can rank candidates differently and recommends reporting both metrics and cleaning choice strings.", "body_md": "Your fine-tune gains three points of `acc_norm`\n\non HellaSwag and loses two points of `acc`\n\n. Same checkpoint, same harness, same seed. Nothing about the model's commonsense reasoning moved in two directions at once — you changed its average per-token entropy, and one of those two metrics is partly measuring how long the answer string is.\n\nThis is the `acc vs acc_norm`\n\nproblem, and it decides which number goes in your model card.\n\n`lm-eval-harness`\n\nmultiple-choice tasks, `acc`\n\npicks the candidate with the highest `acc_norm`\n\ndivides that sum by the `-(length) x (average per-token entropy)`\n\n, so it structurally prefers `acc`\n\nis measuring length as much as knowledge.`acc`\n\nand `acc_norm`\n\nproduce identical rankings.`acc`\n\nand `acc_norm`\n\nin opposite directions with no capability change. Report both, and pick one metric per task family before you start training.Both are argmax over candidate continuations. Given a context `x`\n\nand candidates `y_1 ... y_k`\n\n, the harness runs one forward pass per candidate and computes the summed token log-probability of the continuation:\n\n```\nscore_raw(y) = sum_t log p(y_t | x, y_<t)\n```\n\n`acc`\n\nis `argmax_y score_raw(y)`\n\n. `acc_norm`\n\nis `argmax_y score_raw(y) / len(y.encode(\"utf-8\"))`\n\n.\n\nThat is the whole difference: a denominator. It matters enormously because the numerator is an extensive quantity — it grows with length — while the thing you want to measure is not.\n\nBecause every additional token adds another negative term. For a token sequence of length `L`\n\n, the expected summed log-probability under the model's own distribution is `-L x H`\n\n, where `H`\n\nis the average per-token entropy the model assigns along that path. Well-calibrated English text under an 8B model sits somewhere in the low single digits of nats per token. Adding ten tokens to a candidate costs on the order of ten to twenty nats — far more than the few nats of signal that distinguish a plausible ending from an implausible one.\n\nSo on any task where the correct answer is not systematically the shortest, `acc`\n\nthrows away accuracy for free. Here is the shape of it, with illustrative numbers (not measured — the point is the ordering, not the magnitudes):\n\n| candidate | tokens | bytes | sum logprob | per-token | per-byte |\n|---|---|---|---|---|---|\n| A: \" He leaves.\" (wrong) | 4 | 11 | -8.0 |\n-2.00 | -0.727 |\n| B: \" He picks up the towel and folds it neatly.\" (correct) | 11 | 42 | -14.5 | -1.32 | -0.345 |\n| C: \" He defenestrates it.\" (wrong, rare word) | 9 | 21 | -11.0 | -1.22 |\n-0.524 |\n\nRaw sum picks A. Per-byte picks B. Per-token picks C. Three defensible-looking scoring rules, three different answers, one forward pass each.\n\nBecause token counts are a property of the tokenizer, not the text. Candidate C above is the failure mode: a rare word explodes into many subword pieces, and each of those pieces is highly predictable *given the previous piece*. `defenestr`\n\n-> `ates`\n\nis nearly free. So the sum is spread over a large denominator of cheap tokens, and per-token normalization hands a rare, wrong answer an artificial win.\n\nByte length is tokenizer-independent. That is the real argument for it: it lets you put a Llama-tokenized model and a Gemma-tokenized model in the same table without the fertility difference leaking into the metric. It is not a principled information-theoretic correction — bits-per-byte is a proper cross-tokenizer measure, but `acc_norm`\n\nis applying it as a heuristic tiebreaker over candidates, not as a likelihood.\n\nPractical consequence: the denominator is computed on the **raw string as it appears in the dataset**. Trailing whitespace, a period the annotator forgot, a leading space that your prompt template does or does not consume — each of those shifts the denominator by a byte or two. For a five-character candidate, one byte is a 20% change in the normalized score. Clean your choice strings.\n\nThree scorings from one forward pass:\n\n``` python\nimport torch, torch.nn.functional as F\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"meta-llama/Llama-3.1-8B\"\ntok = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_id, torch_dtype=torch.bfloat16, device_map=\"cuda\"\n).eval()\n\n@torch.no_grad()\ndef score(context: str, continuation: str) -> dict:\n    ctx_ids  = tok(context, add_special_tokens=True).input_ids\n    full_ids = tok(context + continuation, add_special_tokens=True).input_ids\n    n_cont   = len(full_ids) - len(ctx_ids)\n\n    inp = torch.tensor([full_ids], device=model.device)\n    logits = model(inp).logits[0].float()\n    logprobs = F.log_softmax(logits[:-1], dim=-1)      # position i predicts token i+1\n    tok_ll = logprobs.gather(-1, inp[0, 1:].unsqueeze(-1)).squeeze(-1)\n\n    cont_ll = tok_ll[len(ctx_ids) - 1:]                # logprobs of continuation tokens\n    total = cont_ll.sum().item()\n    return {\n        \"acc\":        total,                                    # raw sum\n        \"per_token\":  total / n_cont,\n        \"acc_norm\":   total / len(continuation.encode(\"utf-8\")), # byte-normalized\n    }\n\ndef predict(context, choices, metric=\"acc_norm\"):\n    return max(range(len(choices)),\n               key=lambda i: score(context, choices[i])[metric])\n```\n\nOne caveat that bites people writing their own harness: splitting on `len(ctx_ids)`\n\nassumes tokenizing `context + continuation`\n\nreproduces the context tokens exactly. It usually does when the continuation starts with a space and the context ends with a non-space, and it silently does not when the boundary falls inside a merge. The harness handles this by encoding the pair and aligning; if you roll your own, assert that `full_ids[:len(ctx_ids)] == ctx_ids`\n\nand fail loudly.\n\nBecause MMLU as scored in the harness uses single-letter continuations — `doc_to_choice: [\"A\", \"B\", \"C\", \"D\"]`\n\n. Every candidate is one character, so every denominator is 1, so normalization is a no-op and the argmax is identical. The two reported numbers are the same number.\n\nThis is why \"MMLU is length-robust\" and \"HellaSwag needs `acc_norm`\n\n\" are both true and not in tension. The difference is the answer *format*, not the benchmark's difficulty. Any task you author with letter-key answers is immune; any task that scores full natural-language continuations is exposed.\n\nIt also means you cannot compare a letter-key MMLU run against a full-text-continuation MMLU variant. They are different measurements sharing a name.\n\nBecause SFT and preference optimization change per-token entropy, and the length penalty in raw `acc`\n\nscales with entropy. An instruction-tuned model is sharper on fluent, in-distribution continuations than the base model was. Sharper means each token costs fewer nats, which means the per-token penalty on long candidates shrinks, which means raw `acc`\n\nmechanically drifts up on tasks with long correct answers — with zero change in what the model knows.\n\nThe same effect runs backwards when your SFT data pushes the model toward a format the eval candidates do not match. Now the candidates are out-of-distribution, entropy rises, long answers get hammered, `acc`\n\ncollapses, and `acc_norm`\n\nbarely moves because the denominator absorbs part of it.\n\nRule of thumb: if `acc`\n\nand `acc_norm`\n\nmove in the same direction, you probably changed capability. If they diverge, you probably changed the output distribution's sharpness or format. Check with a small run on a letter-key version of the same questions — if that number is flat, it was never capability.\n\nWhen candidate answers differ in raw frequency more than in length. Byte normalization does nothing about a candidate that is simply a more common string in pretraining data. The fix from the GPT-3 era is unconditional (pointwise-mutual-information style) normalization: score the continuation against a null context and subtract.\n\n```\nscore_pmi(y) = log p(y | x) - log p(y | \"Answer:\")\n```\n\nThis costs a second forward pass per candidate — often cacheable, since the null context is shared — and it targets a genuinely different bias. Use it on tasks whose distractors are high-frequency phrases (many ARC-Easy items) and where length variance is small. Do not stack it blindly on top of byte normalization; you will double-correct and the ranking becomes hard to reason about.\n\nFor a custom multiple-choice task, `output_type: multiple_choice`\n\ncomputes both metrics; you choose which to list:\n\n```\ntask: my_mc_task\ndataset_path: json\ndataset_kwargs:\n  data_files:\n    test: data/mc.jsonl\ntest_split: test\noutput_type: multiple_choice\ndoc_to_text: \"Question: {{question}}\\nAnswer:\"\ndoc_to_choice: \"{{choices}}\"          # e.g. [\" Paris\", \" Lyon\", \" Marseille\"]\ndoc_to_target: \"{{label}}\"            # integer index\nmetric_list:\n  - metric: acc\n    aggregation: mean\n    higher_is_better: true\n  - metric: acc_norm\n    aggregation: mean\n    higher_is_better: true\n```\n\nTwo details worth enforcing in review. First, `doc_to_text`\n\nends with `Answer:`\n\nand every choice starts with a space — keep the space on the choice side, consistently, or your denominators and your tokenization both wobble per-item. Second, if your choices are letter keys, drop `acc_norm`\n\nfrom the list entirely; reporting a metric that is provably identical to another one just invites someone to quote whichever is higher.\n\nA third, unrelated meaning of \"normalized\" shows up at the leaderboard level: rescaling a task's score against its random baseline so a 4-way multiple-choice task does not start at 25%. That happens after aggregation and has nothing to do with `acc_norm`\n\n. Do not conflate them in a results table.\n\n`acc`\n\nand `acc_norm`\n\ndisagree because summed log-likelihood is an extensive quantity that grows with continuation length, so raw `acc`\n\nsystematically favors short candidates, while `acc_norm`\n\ndivides by the continuation's byte length to remove most of that length bias. Use `acc_norm`\n\nfor any task scoring full natural-language continuations of varying length (HellaSwag, ARC, PIQA), where it is the more honest measure of what the model knows; use plain `acc`\n\nfor letter-key formats like MMLU, where the two are numerically identical anyway; reach for PMI-style unconditional normalization when the confound is candidate frequency rather than candidate length. Fix your choice per task family before training starts, report both numbers, and treat a divergence between them as a signal that your model's output distribution changed shape — not that it got smarter.", "url": "https://wpnews.pro/news/acc-vs-acc-norm-why-length-bias-skews-llm-eval-scores", "canonical_source": "https://dev.to/ji_ai/acc-vs-accnorm-why-length-bias-skews-llm-eval-scores-p6e", "published_at": "2026-08-20 15:20:49+00:00", "updated_at": "2026-08-20 15:45:02.212563+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "developer-tools"], "entities": ["lm-eval-harness", "HellaSwag"], "alternates": {"html": "https://wpnews.pro/news/acc-vs-acc-norm-why-length-bias-skews-llm-eval-scores", "markdown": "https://wpnews.pro/news/acc-vs-acc-norm-why-length-bias-skews-llm-eval-scores.md", "text": "https://wpnews.pro/news/acc-vs-acc-norm-why-length-bias-skews-llm-eval-scores.txt", "jsonld": "https://wpnews.pro/news/acc-vs-acc-norm-why-length-bias-skews-llm-eval-scores.jsonld"}}