{"slug": "llm-as-judge-position-bias-swapping-a-b-flipped-87-of-300-verdicts", "title": "LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts", "summary": "A developer found that an LLM-as-judge eval showing a new system prompt winning 66% of 300 head-to-head comparisons dropped to 43% after simply swapping which answer occupied slot A, with slot A winning 369 of 600 judgments (61.5%). A mirror test of 50 identical-text pairs drew 34 picks for A, 9 for B and 7 ties, prompting the developer to recommend judging every pair in both orders and counting a win only when both runs agree.", "body_md": "In a head-to-head eval, my new system prompt beat the old one in 66% of comparisons. I already had the \"we're shipping v2\" message typed.\n\nThen I swapped which answer went first and reran the exact same 300 pairs. The new prompt won 43%.\n\nThe judge, the answers, the rubric and temperature 0 were all the same. The only change was which answer sat in the slot labeled \"A\". That is **LLM-as-judge position bias**, and if your eval script calls `judge(new, old)` in a fixed order, your headline number is probably measuring it too.\n\nLLM-as-judge position bias is a judge model's tendency to pick an answer because of where it appears in the prompt, not because of what it says. It's one of the first biases documented for LLM judges (the MT-Bench / Chatbot Arena paper from Zheng et al. called it out back in 2023), and it's still easy to walk straight into.\n\nIt hits pairwise evals hardest, the \"here are Answer A and Answer B, which is better?\" setup. That's the setup most of us use to compare prompts, models or RAG configs, because it's easier than defining an absolute 1-10 scale.\n\nMy eval script looked like every eval script:\n\n```\nfor q in questions:\n    verdict = ask_judge(q.text, new_answer[q.id], old_answer[q.id])\n    wins += verdict == \"A\"\n```\n\nThe new answer always went in slot A. I didn't pick that order for any reason. It was just the order of the function arguments.\n\nA coworker asked \"did you swap them?\" and I had no good answer. So I ran it again with the old answer in slot A. Here is what the 300 pairs did across both runs:\n\n| Outcome across both orders | Pairs | \n|---|---|\n| New prompt won both times | 120 | \n| Old prompt won both times | 93 | \n| Slot A won both times (verdict flipped) | 78 | \n| Slot B won both times (verdict flipped) | 9 | \n\nCount the slot picks: slot A won 369 of 600 judgments, or 61.5%. A judge with no position preference would land somewhere near 50%.\n\nThe 66% headline was really 120 consistent wins plus 78 pairs where the new prompt just happened to sit in the favored seat.\n\nAn LLM judge prefers the first answer mostly because of how it reads: one token at a time, left to right. Answer A gets evaluated with nothing to compare it to. By the time the model reaches Answer B, it has already formed a picture of what \"good\" looks like for this question, and that picture came from A.\n\nA few things stack on top of that:\n\nNone of this means the judge is useless. On the 213 pairs where it agreed with itself, its picks matched mine on almost every one I spot-checked. The judge is fine at clear differences. On near-ties it isn't measuring quality at all.\n\nRun the mirror test: build pairs where Answer A and Answer B are the exact same text, then ask the judge to pick. A fair judge has no basis to prefer either, so it should say TIE or split close to 50/50.\n\nI ran 50 mirror pairs. My judge picked A 34 times, B 9 times and TIE 7 times. It was confidently picking a winner between two copies of the same text, and it usually picked the first one.\n\nThis takes ten minutes and tells you more about your judge than a week of reading verdicts. I now run it every time I change the judge model or the judge prompt.\n\nJudge every pair twice, once in each order, and count a win only when both runs agree. If the judge contradicts itself, call it a tie. That's the whole fix:\n\n```\nJUDGE_PROMPT = \"\"\"You are comparing two answers to the same question.\n\nQuestion:\n{question}\n\nAnswer A:\n{a}\n\nAnswer B:\n{b}\n\nBriefly list the strengths and weaknesses of each answer.\nThen end with exactly one line: VERDICT: A, VERDICT: B, or VERDICT: TIE\"\"\"\n\ndef ask_judge(question, a, b):\n    out = call_llm(JUDGE_PROMPT.format(question=question, a=a, b=b), temperature=0)\n    return parse_verdict(out)  # \"A\" | \"B\" | \"TIE\"\n\ndef judge_pair(question, new, old):\n    run1 = ask_judge(question, new, old)  # new in slot A\n    run2 = ask_judge(question, old, new)  # new in slot B\n\n    if run1 == \"A\" and run2 == \"B\":\n        return \"new\"\n    if run1 == \"B\" and run2 == \"A\":\n        return \"old\"\n    # Self-contradiction, or a TIE in either run: no signal.\n    return \"tie\"\n```\n\nThen report three numbers instead of one:\n\n```\nresults = [judge_pair(q.text, new[q.id], old[q.id]) for q in questions]\nn = len(results)\n\nnew_wins = results.count(\"new\")\nold_wins = results.count(\"old\")\nties = results.count(\"tie\")\n\nprint(f\"new {new_wins/n:.0%}  old {old_wins/n:.0%}  tie {ties/n:.0%}\")\nprint(f\"new win rate on decisive pairs: {new_wins/(new_wins+old_wins):.0%}\")\n```\n\nThe tie rate is the important number here. It tells you how much of your eval set the judge can't actually separate. If it's 30%, a 5-point swing in your headline doesn't mean much.\n\nSome other things that helped:\n\nRandomizing the order per pair is better than a fixed order, but it doesn't fix the problem. Random assignment spreads the position bias evenly across both systems, so your aggregate win rate stops being skewed in one direction. That's real progress.\n\nWhat it hides is the noise. Those 87 coin-flip pairs are still in your data, now split randomly between \"new won\" and \"old won.\" Your headline drifts toward 50% and looks like an honest close race, when actually a third of your eval set says nothing at all. Rerun it with a different random seed and the number moves.\n\nRunning both orders costs twice the judge calls. On 300 pairs, that's 300 extra API calls. Shipping a prompt that was never better costs more.\n\nPosition bias rarely travels alone. Watch for these two:\n\nSwapping the order fixes neither of these. They need their own checks.\n\nYes, a little. On the 213 pairs where the judge agreed with itself, the new prompt won 120 to 93, which is 56%. That's a real but modest improvement. It isn't the 66% blowout I nearly announced. I shipped it with a much quieter message and a note that 29% of the eval set was a toss-up.\n\nLLM-as-judge position bias makes a pairwise judge favor whichever answer occupies a particular slot, usually the first. It mostly decides near-tie comparisons, which can quietly inflate or deflate a win rate by double digits. In my eval, swapping A and B flipped 87 of 300 verdicts and turned a 66% win into 43%. The fix is to judge every pair in both orders, count only verdicts that agree, treat contradictions as ties, and report the tie rate next to the win rate. Run the mirror test, with the identical answer in both slots, before you trust any judge.\n\n*Written by the developer behind [Preterview](https://preterview.com/en), an interview prep platform.*", "url": "https://wpnews.pro/news/llm-as-judge-position-bias-swapping-a-b-flipped-87-of-300-verdicts", "canonical_source": "https://dev.to/ji_ai/llm-as-judge-position-bias-swapping-ab-flipped-87-of-300-verdicts-42hf", "published_at": "2026-11-01 12:15:13+00:00", "updated_at": "2026-11-01 12:22:24.625450+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools", "mlops"], "entities": ["Zheng et al.", "MT-Bench", "Chatbot Arena"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llm-as-judge-position-bias-swapping-a-b-flipped-87-of-300-verdicts", "markdown": "https://wpnews.pro/news/llm-as-judge-position-bias-swapping-a-b-flipped-87-of-300-verdicts.md", "text": "https://wpnews.pro/news/llm-as-judge-position-bias-swapping-a-b-flipped-87-of-300-verdicts.txt", "jsonld": "https://wpnews.pro/news/llm-as-judge-position-bias-swapping-a-b-flipped-87-of-300-verdicts.jsonld"}}