Your fine-tune gains three points of acc_norm
on HellaSwag and loses two points of acc
. Same checkpoint, same harness, same seed. Nothing about the model's commonsense reasoning moved in two directions at once — you changed its average per-token entropy, and one of those two metrics is partly measuring how long the answer string is.
This is the acc vs acc_norm
problem, and it decides which number goes in your model card.
lm-eval-harness
multiple-choice tasks, acc
picks the candidate with the highest acc_norm
divides that sum by the -(length) x (average per-token entropy)
, so it structurally prefers acc
is measuring length as much as knowledge.acc
and acc_norm
produce identical rankings.acc
and acc_norm
in opposite directions with no capability change. Report both, and pick one metric per task family before you start training.Both are argmax over candidate continuations. Given a context x
and candidates y_1 ... y_k
, the harness runs one forward pass per candidate and computes the summed token log-probability of the continuation:
score_raw(y) = sum_t log p(y_t | x, y_<t)
acc
is argmax_y score_raw(y)
. acc_norm
is argmax_y score_raw(y) / len(y.encode("utf-8"))
.
That is the whole difference: a denominator. It matters enormously because the numerator is an extensive quantity — it grows with length — while the thing you want to measure is not.
Because every additional token adds another negative term. For a token sequence of length L
, the expected summed log-probability under the model's own distribution is -L x H
, where H
is the average per-token entropy the model assigns along that path. Well-calibrated English text under an 8B model sits somewhere in the low single digits of nats per token. Adding ten tokens to a candidate costs on the order of ten to twenty nats — far more than the few nats of signal that distinguish a plausible ending from an implausible one.
So on any task where the correct answer is not systematically the shortest, acc
throws away accuracy for free. Here is the shape of it, with illustrative numbers (not measured — the point is the ordering, not the magnitudes):
| candidate | tokens | bytes | sum logprob | per-token | per-byte |
|---|---|---|---|---|---|
| A: " He leaves." (wrong) | 4 | 11 | -8.0 | ||
| -2.00 | -0.727 | ||||
| B: " He picks up the towel and folds it neatly." (correct) | 11 | 42 | -14.5 | -1.32 | -0.345 |
| C: " He defenestrates it." (wrong, rare word) | 9 | 21 | -11.0 | -1.22 | |
| -0.524 |
Raw sum picks A. Per-byte picks B. Per-token picks C. Three defensible-looking scoring rules, three different answers, one forward pass each.
Because token counts are a property of the tokenizer, not the text. Candidate C above is the failure mode: a rare word explodes into many subword pieces, and each of those pieces is highly predictable given the previous piece. defenestr
-> ates
is nearly free. So the sum is spread over a large denominator of cheap tokens, and per-token normalization hands a rare, wrong answer an artificial win.
Byte length is tokenizer-independent. That is the real argument for it: it lets you put a Llama-tokenized model and a Gemma-tokenized model in the same table without the fertility difference leaking into the metric. It is not a principled information-theoretic correction — bits-per-byte is a proper cross-tokenizer measure, but acc_norm
is applying it as a heuristic tiebreaker over candidates, not as a likelihood.
Practical consequence: the denominator is computed on the raw string as it appears in the dataset. Trailing whitespace, a period the annotator forgot, a leading space that your prompt template does or does not consume — each of those shifts the denominator by a byte or two. For a five-character candidate, one byte is a 20% change in the normalized score. Clean your choice strings.
Three scorings from one forward pass:
import torch, torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "meta-llama/Llama-3.1-8B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="cuda"
).eval()
@torch.no_grad()
def score(context: str, continuation: str) -> dict:
ctx_ids = tok(context, add_special_tokens=True).input_ids
full_ids = tok(context + continuation, add_special_tokens=True).input_ids
n_cont = len(full_ids) - len(ctx_ids)
inp = torch.tensor([full_ids], device=model.device)
logits = model(inp).logits[0].float()
logprobs = F.log_softmax(logits[:-1], dim=-1) # position i predicts token i+1
tok_ll = logprobs.gather(-1, inp[0, 1:].unsqueeze(-1)).squeeze(-1)
cont_ll = tok_ll[len(ctx_ids) - 1:] # logprobs of continuation tokens
total = cont_ll.sum().item()
return {
"acc": total, # raw sum
"per_token": total / n_cont,
"acc_norm": total / len(continuation.encode("utf-8")), # byte-normalized
}
def predict(context, choices, metric="acc_norm"):
return max(range(len(choices)),
key=lambda i: score(context, choices[i])[metric])
One caveat that bites people writing their own harness: splitting on len(ctx_ids)
assumes tokenizing context + continuation
reproduces the context tokens exactly. It usually does when the continuation starts with a space and the context ends with a non-space, and it silently does not when the boundary falls inside a merge. The harness handles this by encoding the pair and aligning; if you roll your own, assert that full_ids[:len(ctx_ids)] == ctx_ids
and fail loudly.
Because MMLU as scored in the harness uses single-letter continuations — doc_to_choice: ["A", "B", "C", "D"]
. Every candidate is one character, so every denominator is 1, so normalization is a no-op and the argmax is identical. The two reported numbers are the same number.
This is why "MMLU is length-robust" and "HellaSwag needs acc_norm
" are both true and not in tension. The difference is the answer format, not the benchmark's difficulty. Any task you author with letter-key answers is immune; any task that scores full natural-language continuations is exposed.
It also means you cannot compare a letter-key MMLU run against a full-text-continuation MMLU variant. They are different measurements sharing a name.
Because SFT and preference optimization change per-token entropy, and the length penalty in raw acc
scales with entropy. An instruction-tuned model is sharper on fluent, in-distribution continuations than the base model was. Sharper means each token costs fewer nats, which means the per-token penalty on long candidates shrinks, which means raw acc
mechanically drifts up on tasks with long correct answers — with zero change in what the model knows.
The same effect runs backwards when your SFT data pushes the model toward a format the eval candidates do not match. Now the candidates are out-of-distribution, entropy rises, long answers get hammered, acc
collapses, and acc_norm
barely moves because the denominator absorbs part of it.
Rule of thumb: if acc
and acc_norm
move in the same direction, you probably changed capability. If they diverge, you probably changed the output distribution's sharpness or format. Check with a small run on a letter-key version of the same questions — if that number is flat, it was never capability.
When candidate answers differ in raw frequency more than in length. Byte normalization does nothing about a candidate that is simply a more common string in pretraining data. The fix from the GPT-3 era is unconditional (pointwise-mutual-information style) normalization: score the continuation against a null context and subtract.
score_pmi(y) = log p(y | x) - log p(y | "Answer:")
This costs a second forward pass per candidate — often cacheable, since the null context is shared — and it targets a genuinely different bias. Use it on tasks whose distractors are high-frequency phrases (many ARC-Easy items) and where length variance is small. Do not stack it blindly on top of byte normalization; you will double-correct and the ranking becomes hard to reason about.
For a custom multiple-choice task, output_type: multiple_choice
computes both metrics; you choose which to list:
task: my_mc_task
dataset_path: json
dataset_kwargs:
data_files:
test: data/mc.jsonl
test_split: test
output_type: multiple_choice
doc_to_text: "Question: {{question}}\nAnswer:"
doc_to_choice: "{{choices}}" # e.g. [" Paris", " Lyon", " Marseille"]
doc_to_target: "{{label}}" # integer index
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- metric: acc_norm
aggregation: mean
higher_is_better: true
Two details worth enforcing in review. First, doc_to_text
ends with Answer:
and every choice starts with a space — keep the space on the choice side, consistently, or your denominators and your tokenization both wobble per-item. Second, if your choices are letter keys, drop acc_norm
from the list entirely; reporting a metric that is provably identical to another one just invites someone to quote whichever is higher.
A third, unrelated meaning of "normalized" shows up at the leaderboard level: rescaling a task's score against its random baseline so a 4-way multiple-choice task does not start at 25%. That happens after aggregation and has nothing to do with acc_norm
. Do not conflate them in a results table.
acc
and acc_norm
disagree because summed log-likelihood is an extensive quantity that grows with continuation length, so raw acc
systematically favors short candidates, while acc_norm
divides by the continuation's byte length to remove most of that length bias. Use acc_norm
for any task scoring full natural-language continuations of varying length (HellaSwag, ARC, PIQA), where it is the more honest measure of what the model knows; use plain acc
for letter-key formats like MMLU, where the two are numerically identical anyway; reach for PMI-style unconditional normalization when the confound is candidate frequency rather than candidate length. Fix your choice per task family before training starts, report both numbers, and treat a divergence between them as a signal that your model's output distribution changed shape — not that it got smarter.