LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts

In a head-to-head eval, my new system prompt beat the old one in 66% of comparisons. I already had the "we're shipping v2" message typed.

Then I swapped which answer went first and reran the exact same 300 pairs. The new prompt won 43%.

The judge, the answers, the rubric and temperature 0 were all the same. The only change was which answer sat in the slot labeled "A". That is LLM-as-judge position bias, and if your eval script calls judge(new, old) in a fixed order, your headline number is probably measuring it too.

LLM-as-judge position bias is a judge model's tendency to pick an answer because of where it appears in the prompt, not because of what it says. It's one of the first biases documented for LLM judges (the MT-Bench / Chatbot Arena paper from Zheng et al. called it out back in 2023), and it's still easy to walk straight into.

It hits pairwise evals hardest, the "here are Answer A and Answer B, which is better?" setup. That's the setup most of us use to compare prompts, models or RAG configs, because it's easier than defining an absolute 1-10 scale.

My eval script looked like every eval script:

for q in questions:
    verdict = ask_judge(q.text, new_answer[q.id], old_answer[q.id])
    wins += verdict == "A"

The new answer always went in slot A. I didn't pick that order for any reason. It was just the order of the function arguments.

A coworker asked "did you swap them?" and I had no good answer. So I ran it again with the old answer in slot A. Here is what the 300 pairs did across both runs:

Count the slot picks: slot A won 369 of 600 judgments, or 61.5%. A judge with no position preference would land somewhere near 50%.

The 66% headline was really 120 consistent wins plus 78 pairs where the new prompt just happened to sit in the favored seat.

An LLM judge prefers the first answer mostly because of how it reads: one token at a time, left to right. Answer A gets evaluated with nothing to compare it to. By the time the model reaches Answer B, it has already formed a picture of what "good" looks like for this question, and that picture came from A.

A few things stack on top of that:

None of this means the judge is useless. On the 213 pairs where it agreed with itself, its picks matched mine on almost every one I spot-checked. The judge is fine at clear differences. On near-ties it isn't measuring quality at all.

Run the mirror test: build pairs where Answer A and Answer B are the exact same text, then ask the judge to pick. A fair judge has no basis to prefer either, so it should say TIE or split close to 50/50.

I ran 50 mirror pairs. My judge picked A 34 times, B 9 times and TIE 7 times. It was confidently picking a winner between two copies of the same text, and it usually picked the first one.

This takes ten minutes and tells you more about your judge than a week of reading verdicts. I now run it every time I change the judge model or the judge prompt.

Judge every pair twice, once in each order, and count a win only when both runs agree. If the judge contradicts itself, call it a tie. That's the whole fix:

JUDGE_PROMPT = """You are comparing two answers to the same question.

Question:
{question}

Answer A:
{a}

Answer B:
{b}

Briefly list the strengths and weaknesses of each answer.
Then end with exactly one line: VERDICT: A, VERDICT: B, or VERDICT: TIE"""

def ask_judge(question, a, b):
    out = call_llm(JUDGE_PROMPT.format(question=question, a=a, b=b), temperature=0)
    return parse_verdict(out)  # "A" | "B" | "TIE"

def judge_pair(question, new, old):
    run1 = ask_judge(question, new, old)  # new in slot A
    run2 = ask_judge(question, old, new)  # new in slot B

    if run1 == "A" and run2 == "B":
        return "new"
    if run1 == "B" and run2 == "A":
        return "old"
    return "tie"

Then report three numbers instead of one:

results = [judge_pair(q.text, new[q.id], old[q.id]) for q in questions]
n = len(results)

new_wins = results.count("new")
old_wins = results.count("old")
ties = results.count("tie")

print(f"new {new_wins/n:.0%}  old {old_wins/n:.0%}  tie {ties/n:.0%}")
print(f"new win rate on decisive pairs: {new_wins/(new_wins+old_wins):.0%}")

The tie rate is the important number here. It tells you how much of your eval set the judge can't actually separate. If it's 30%, a 5-point swing in your headline doesn't mean much.

Some other things that helped:

Randomizing the order per pair is better than a fixed order, but it doesn't fix the problem. Random assignment spreads the position bias evenly across both systems, so your aggregate win rate stops being skewed in one direction. That's real progress.

What it hides is the noise. Those 87 coin-flip pairs are still in your data, now split randomly between "new won" and "old won." Your headline drifts toward 50% and looks like an honest close race, when actually a third of your eval set says nothing at all. Rerun it with a different random seed and the number moves.

Running both orders costs twice the judge calls. On 300 pairs, that's 300 extra API calls. Shipping a prompt that was never better costs more.

Position bias rarely travels alone. Watch for these two:

Swapping the order fixes neither of these. They need their own checks.

Yes, a little. On the 213 pairs where the judge agreed with itself, the new prompt won 120 to 93, which is 56%. That's a real but modest improvement. It isn't the 66% blowout I nearly announced. I shipped it with a much quieter message and a note that 29% of the eval set was a toss-up.

LLM-as-judge position bias makes a pairwise judge favor whichever answer occupies a particular slot, usually the first. It mostly decides near-tie comparisons, which can quietly inflate or deflate a win rate by double digits. In my eval, swapping A and B flipped 87 of 300 verdicts and turned a 66% win into 43%. The fix is to judge every pair in both orders, count only verdicts that agree, treat contradictions as ties, and report the tie rate next to the win rate. Run the mirror test, with the identical answer in both slots, before you trust any judge.

Written by the developer behind Preterview, an interview prep platform.