LLM benchmarks: why the leaderboard doesn't predict your agent Six major LLM leaderboards reviewed side by side on November 10, 2026 named four different models as best, with LMArena's text board ranking Gemini 4 Carbon first, the Artificial Analysis Intelligence Index and LiveBench ranking Claude Fable 5.2 first, Terminal-Bench and LMArena's WebDev board ranking Claude Opus 5.5 first, and Sierra's τ³-bench banking board ranking Qwen 3.8 Max first, according to EvalShift developer and post author. The post attributes the disagreement to each score measuring a model inside one harness, on one task set, under one grader, citing GPT-5 scoring 49.6% in one agent harness versus 33.9% in another on Terminal-Bench 2.0, a 6-point swing Anthropic measured from container resources alone, and roughly half of AI patches passing SWE-bench Verified's tests that project maintainers would not merge. The author argues leaderboards are useful only for shortlisting two or three models, not for predicting which one an agent should run. ← all posts https://www.evalshift.dev/blog LLM benchmarks: why the leaderboard doesn't predict your agent What LLM benchmarks measure, where the November 2026 leaderboards disagree on the best model, and how to test a shortlist on your own agent's traffic. An LLM benchmark is a fixed set of tasks with a scoring rule, and an LLM leaderboard is a table that ranks models by it. On November 10, 2026 I read the main ones side by side. LMArena's text board had Gemini 4 Carbon on top. The Artificial Analysis Intelligence Index and LiveBench had Claude Fable 5.2. Terminal-Bench and LMArena's WebDev board had Claude Opus 5.5. Sierra's τ³-bench banking board, the one built around a customer-service agent, had Qwen 3.8 Max. Six boards named four different models as the best one. They disagree because each score measures a model inside one harness, on one set of tasks, under one grader. Your agent shares only the model. On Terminal-Bench 2.0 the same model, GPT-5, scored 49.6% in one agent harness and 33.9% in another. On τ-bench, a model that completes a task once often fails it when asked again. Anthropic measured a 6-point swing on Terminal-Bench 2.0 from container resources alone. And roughly half of the AI patches that pass SWE-bench Verified's tests would not be merged by the projects' own maintainers. A leaderboard is still the right place to start, because it tells you which two or three models are worth testing. It can't tell you which of them your agent should run. The rest of this post covers what the main benchmarks measure, what the boards say about the best LLM for coding and for agents this month, five reasons the scores don't carry over, and how to test a shortlist on your own traffic. I build EvalShift, which automates that last part, but the method works with any tooling. What an LLM benchmark is Every LLM benchmark has three parts: a set of tasks , a harness that puts each task in front of the model the prompt, the tools, the loop, the time and step limits , and a grader that turns the output into a score. A leaderboard publishes the scores as a ranking. Two benchmarks with the same name on the tin, "coding" or "agents", can differ in all three. | Benchmark | Tasks | Graded by | Harness | |---|---|---|---| | MMLU, GPQA Diamond | Multiple-choice questions GPQA Diamond has 198 | Answer key | One prompt per question | | LMArena ↗ https://arena.ai/how-it-works | Whatever its users type | A human vote between two anonymous models | A chat window | | SWE-bench Verified ↗ https://www.swebench.com/verified.html | 500 real GitHub issues | The repository's tests | Any scaffold on the main board; a "bash only" board fixes one | | SWE-bench Pro ↗ https://labs.scale.com/leaderboard/swe bench pro public | 1,865 tasks from 41 repositories, 731 of them public | Tests | Varies by entry | | Terminal-Bench ↗ https://www.tbench.ai/leaderboard | Containerized terminal tasks | A verifier script | Each row is a model paired with an agent | | Aider polyglot ↗ https://aider.chat/docs/leaderboards/ | 225 Exercism exercises in six languages | Tests | Aider | | τ-bench ↗ https://taubench.com/ | Customer-service conversations with a simulated user, tools and a policy | The final database state, over repeated trials | Sierra's scaffold | | GAIA ↗ https://huggingface.co/spaces/gaia-benchmark/leaderboard | More than 450 questions with one right answer | Exact answer | Whole agent systems are submitted | | OSWorld-Verified ↗ https://os-world.github.io/ | 369 desktop tasks | Execution-based scripts | Run by the OSWorld team | | Artificial Analysis Intelligence Index ↗ https://artificialanalysis.ai/methodology/intelligence-benchmarking | Nine evals in four equally weighted categories | Mixed | Artificial Analysis runs everything itself | | LiveBench ↗ https://livebench.ai/ /methodology | Seven suites, five of them agentic | Scripts and rubrics | One run per model | | BenchLM ↗ https://benchlm.ai/methodology | None of its own | Other people's benchmarks, calibrated onto one scale | None | Three things to notice before reading any ranking: - + The oldest kind is saturated. PhD-level experts score about 65% on GPQA Diamond. The top entry on Artificial Analysis's GPQA page ↗ https://artificialanalysis.ai/evaluations/gpqa-diamond is 96.3%, and the next two are at 96.1% and 95.3%. A benchmark where the frontier models all score in the mid-90s no longer separates them. - + "Agentic" boards rank systems, not models. A Terminal-Bench row is a model plus an agent. GAIA's top three entries this month are all multi-model harnesses. The score belongs to the pair. - + Composite indexes are weighted averages of the above. LiveBench's overall score counts its long agentic suite twice and its coding and design suites one and a half times. BenchLM runs no evals at all and labels its top two rows "Estimated". A composite inherits every limit of its parts and adds a choice of weights. The LLM leaderboard in November 2026: the best model depends on the board Here is the top entry on each board as I read it on November 10, 2026. Scores are as printed, with the board's own error bars where it shows them. | Leaderboard | Top entry | Score | Runner-up | |---|---|---|---| | LMArena, text | Gemini 4 Carbon High , marked Preliminary | 1526 ±10 | Gemini 4 Argon High , 1522 ±5 | | Artificial Analysis Intelligence Index v5.0.1 | Claude Fable 5.2 max | 65 | Gemini 4 Carbon high , 60, after two more Fable 5.2 settings at 62 | | LiveBench, overall | Claude Fable 5.2 xhigh | 84.9 | GPT-6 Astra, 84.5 | | LiveBench, coding | Claude Fable 5.1 | 89.7 | GPT-6.1 Terra, 89.2 | | LMArena, WebDev | Claude Opus 5.5 Max | 1818 ±12 | Claude Sonnet 5.5 xHigh , 1774 ±10 | | Terminal-Bench 4.1 | Claude Opus 5.5 max + Claude Code | 65.8% ±3.1 | Claude Sonnet 5.5 max + Claude Code, 63.9% ±3.0 | | LiveBench, agentic | Claude Fable 5.2 | 82.9 | GPT-6 Astra, 75.6 | | OSWorld-Verified | Claude Fable 5 | 86.0% | Claude Opus 5, 83.4% | | τ³-bench, banking | Qwen 3.8 Max | 55.2% | Claude Opus 5, 48.7% | | GAIA, test set | openJiuwen-deepagent and Keen AgenticOS V1.1, tied | 94.35 | Shadow Harness V8.01, 94.02 | Sources: LMArena text ↗ https://arena.ai/leaderboard/text and WebDev ↗ https://arena.ai/leaderboard/webdev , Artificial Analysis ↗ https://artificialanalysis.ai/leaderboards/models , LiveBench ↗ https://livebench.ai/ , Terminal-Bench ↗ https://www.tbench.ai/leaderboard , OSWorld ↗ https://os-world.github.io/ , τ-bench ↗ https://taubench.com/ and GAIA ↗ https://huggingface.co/spaces/gaia-benchmark/leaderboard . Best LLM for coding By these boards, the best LLM for coding in November 2026 is Claude Opus 5.5, Claude Fable 5.1 or Claude Fable 5.2, depending on which one you open. Opus 5.5 leads Terminal-Bench 4.1 and LMArena WebDev. Fable 5.1 leads LiveBench's coding column at 89.7, where Fable 5.2, the model that leads LiveBench overall, scores 79.2. BenchLM's coding ranking ↗ https://benchlm.ai/coding puts Fable 5.2 first. The best-known coding leaderboards can't settle it, because none of them lists those three models. SWE-bench Verified's newest entry is dated February 26, 2026, and its top row is Claude 4.5 Opus with a third-party agent at 79.2%. SWE-bench Pro's public board is led by Muse Spark 1.1 at 61.5%. Aider's polyglot board was last updated in November 2025. A leaderboard only ranks the models someone has run on it. Best LLM for agents For agents the boards agree less, because "agent" means a different job on each. LiveBench's agentic suite and BenchLM's agentic ranking ↗ https://benchlm.ai/agentic put Claude Fable 5.2 first. OSWorld-Verified, which is desktop computer use, has Claude Fable 5. τ³-bench banking, where the agent talks to a customer, looks up policy and calls tools, has Qwen 3.8 Max 6.5 points ahead of the next model. GAIA's top entries are harnesses that combine several models, so its ranking says little about any one of them. Two more things in that table matter as much as the names: - + Most gaps at the top are inside the error bars. LMArena's text leader is 4 points ahead with a ±10 interval. Terminal-Bench's top two are 1.9 points apart with ±3 on each. LiveBench's top three are within 0.9 points of each other, and its methodology page says each score comes from a single run. - + The same model changes rank between columns of one board. Fable 5.2 is first on LiveBench overall and more than 10 points behind the leader on its coding suite. "Best overall" is an average, and your agent doesn't run the average. Why an LLM benchmark score doesn't carry over to your agent Five reasons, each with published numbers. 1. The score belongs to a model and a harness, and you run a different harness The Terminal-Bench 2.0 paper ↗ https://arxiv.org/abs/2601.11868 ran the same models inside several agent harnesses, at least five times each: | Model | Best harness | Worst harness | Spread | |---|---|---|---| | GPT-5 | Codex CLI, 49.6% | Mini-SWE-Agent, 33.9% | 15.7 points | | Gemini 2.5 Pro | Terminus 2, 32.6% | OpenHands, 15.7% | 16.9 points | | Claude Opus 4.5 | Terminus 2, 57.8% | OpenHands, 51.9% | 5.9 points | | Claude Sonnet 4.5 | Terminus 2, 42.8% | Claude Code, 40.1% | 2.7 points | Which model wins depends on the harness. Under Codex CLI, GPT-5 beats every Claude Sonnet 4.5 row. Under Terminus 2, Sonnet 4.5 beats GPT-5 by 7.6 points. The paper's own reading is that the model usually matters more than the scaffold, and across the whole table that holds. For one model in one harness, it can be the smaller effect. The harness can be tuned with the model held fixed. LangChain ↗ https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering took its coding agent from 52.8 to 66.5 on Terminal-Bench 2.0 by changing only the harness around gpt-5.2-codex . The Princeton-led team behind the Holistic Agent Leaderboard ↗ https://arxiv.org/abs/2510.11977 ran 21,730 agent rollouts across 9 models and 9 benchmarks, found that scaffolds change both accuracy and cost, and found that higher reasoning effort did not improve accuracy in 21 of 36 runs. Your harness is your system prompt, your tool definitions and your loop, and no leaderboard has run it. Small parts of it move results. IBM researchers rephrased the user requests ↗ https://arxiv.org/abs/2504.00914 in a function-calling benchmark without changing their meaning: gpt-4o-mini fell from 0.925 to 0.765 on tool-call accuracy and Claude 3.5 Sonnet from 0.915 to 0.845. Anthropic's guide to writing tools for agents ↗ https://www.anthropic.com/engineering/writing-tools-for-agents reports that small edits to tool descriptions produced large changes in its tool-use evals. 2. One success is not reliability Most leaderboards report whether the model completed each task once. A production agent meets the same kind of request thousands of times. τ-bench ↗ https://arxiv.org/abs/2406.12045 introduced pass^k for this: the chance that all k independent attempts at a task succeed. In the original 2024 paper, a gpt-4o function-calling agent completed 61.2% of retail tasks on one attempt and under 25% on all of eight. Newer models have the same gap from a higher starting point. Sierra's leaderboard submissions ↗ https://github.com/sierra-research/tau2-bench/tree/main/web/leaderboard/public/submissions report four trials per task: | Model, domain | 1 of 1 | 4 of 4 | |---|---|---| | Claude Opus 4.5, retail | 79.6% | 51.8% | | Claude Opus 4.5, airline | 84.0% | 70.0% | | Claude Opus 4.5, telecom | 92.3% | 78.1% | The arithmetic is unforgiving: an agent that succeeds 90% of the time on a task succeeds five times in a row 59% of the time. A 2026 Princeton paper, Towards a Science of AI Agent Reliability ↗ https://arxiv.org/abs/2602.16666 , ran 15 models five times each on GAIA and τ-bench and found that reliability has improved more slowly than headline accuracy. A leaderboard column of single-attempt scores can't show you that. 3. The gap between two models is often smaller than the noise Anthropic ran Terminal-Bench 2.0 under different container resource limits and found a 6-point gap ↗ https://www.anthropic.com/engineering/infrastructure-noise between the most and least generous setups, with the model unchanged. Its rule of thumb is to distrust leaderboard differences under 3 points until the configurations are documented and matched. A separate study of 60,000 agent trajectories on SWE-bench Verified On Randomness in Agentic Evals ↗ https://arxiv.org/abs/2602.07150 found that a single-run score moves by 2.2 to 6.0 points depending on which run you happened to get, and that the spread persists at temperature 0. Few benchmarks report this. An Oxford-led review of 445 LLM benchmarks ↗ https://arxiv.org/abs/2511.04703 found that 16% ran any statistical test. Apply a 3-point rule to the November table and first place on LiveBench overall, LiveBench coding, Terminal-Bench, OSWorld-Verified and GAIA becomes a tie. 4. Public tasks leak, and graders have holes A public benchmark's tasks and answers end up in training data, and its grader is only as strict as the tests somebody wrote. - + Contamination. On February 23, 2026 OpenAI stopped reporting SWE-bench Verified ↗ https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ and asked other labs to do the same. It audited 138 problems its model did not solve consistently and found material flaws in the tests or the task description in 59.4% of them. It also found that frontier models from three labs could reproduce the original fixes or task details from memory. - + Private code is harder. SWE-bench Pro ↗ https://arxiv.org/abs/2509.16941v1 has a public set and a commercial set drawn from private startup codebases. In its first release GPT-5 scored 23.3% on the public set and 14.9% on the commercial one, and Claude Opus 4.1 went from 22.7% to 17.8%. The model that was ahead on the public set was behind on the private one. - + Passing the tests is not the same as doing the job. METR asked four maintainers of three SWE-bench repositories to review 296 AI-written pull requests and estimated ↗ https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/ that roughly half of the ones that pass the benchmark's tests would not be merged. METR notes the agents had no chance to respond to review, so this is a gap between the grader and the job, not a ceiling on the models. - + Harnesses can be exploited. In April 2026 a UC Berkeley group published proof-of-concept exploits ↗ https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/ that scored 100% on SWE-bench Verified, SWE-bench Pro and Terminal-Bench, about 98% on GAIA and 73% on OSWorld without solving a task. That shows what the graders would accept. It is not evidence that any leaderboard entry cheated. - + Arena scores can be shopped. The Leaderboard Illusion ↗ https://arxiv.org/abs/2504.20879 documented 27 private model variants tested by one lab on Chatbot Arena before a release, and argued that a lab can publish only the best-scoring one. LMArena's response ↗ https://news.lmarena.ai/our-response disputes the size of the effect and puts it at around 11 Elo points. 5. Your tasks are not the benchmark's tasks A benchmark score is an average over somebody else's task distribution. Your agent has a handful of routes, its own tools, its own policies and its own users. No public board averages over those. The model providers say this themselves. Anthropic's ↗ https://platform.claude.com/docs/en/test-and-evaluate/develop-tests and OpenAI's ↗ https://developers.openai.com/api/docs/guides/evaluation-best-practices evaluation guides both tell teams to build evals that mirror their real task distribution, and Google's ↗ https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview says its evaluation service gives insight that public leaderboards and general benchmarks can't. Databricks benchmarked coding agents on its own codebase ↗ https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase in July 2026 because the public benchmarks couldn't answer its questions, and found an open-source model statistically tied with the top scorer at two thirds of its cost per task. I have seen both outcomes on real agents. I replayed 16 real examples from one agent on its current model and a cheaper candidate: 58.8% cheaper, 75.2% faster, no failed calls, and the candidate picked a different tool on 25% of comparisons against a 10% limit. The report came back FAIL https://www.evalshift.dev/blog/cheaper-faster-and-still-a-fail . A customer-support agent with six tools moved to a cheaper model over 120 production examples and held every budget https://www.evalshift.dev/blog/what-a-passing-migration-proves at 71.2% lower cost. Nothing on a leaderboard told me which of the two would pass. What LLM leaderboards are good for None of this makes leaderboards useless. They do three jobs well: - + Shortlisting. A model 20 points behind on every board that resembles your task is unlikely to win on yours. Leaderboards cut a field of dozens to two or three candidates. - + Rough tiers. A gap much larger than the noise is real information, even when a 2-point gap is not. - + Price and speed. Artificial Analysis measures cost and latency alongside quality, which tells you what a candidate would have to beat to be worth a switch. The cheapest LLM API comparison https://www.evalshift.dev/blog/cheapest-llm-api covers why list price is still not what an agent costs. To read one, ask five questions of the board before you look at the names: 1. + What are the tasks? Find the board whose tasks look most like your agent's. A customer-service agent is closer to τ-bench than to Terminal-Bench. 2. + What harness produced the score? A row that pairs the model with its vendor's own agent tells you about that agent. Look for a board, or a column, that holds the harness fixed. 3. + Who graded it? Tests, an answer key, a human vote and an LLM judge fail in different ways. 4. + How many runs, and is there an error bar? One run with no interval can't separate neighbors. 5. + How old are the tasks, and are they public? Public tasks older than the model's training cutoff may be in its training data. Check the date of the newest entry too. How to pick a model for your agent Six steps. They are the same whether you are choosing a first model or deciding whether a new release should replace the one you run. Step 1: shortlist from the boards closest to your task Pick two or three candidates from the leaderboards whose tasks and harness resemble yours, plus the model you run today as the baseline. Include one model cheaper than you think you need: the passing report above was a 71.2% saving that no ranking would have suggested. Step 2: build the test from your own traffic Take 100 to 300 real inputs from production, stratified by route or intent so the rare and expensive paths are in the sample. For an agent, take whole conversations with the tool results they produced. A golden suite built from production traffic https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic is the reusable version, and it is a benchmark nobody has trained on. Size the routes you care about, not only the total. Zero failures in 20 examples still allows a true failure rate of about 15% at 95% confidence, and zero in 100 brings that to about 3%. How many eval cases you need https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need has the full sizing. Step 3: write the limits before the run For each route, write down the most regressions you will accept, the most tool-selection divergence, and the cost or latency change that would make a switch worth it. Routes that move money or touch security get zero. Limits written after the numbers are in tend to match the numbers. Step 4: run every candidate inside your own harness Run each candidate with your system prompt, your tool definitions and your loop, on the same inputs as the baseline. This is the step a leaderboard can't do for you, and reason 1 above is why it matters. If a candidate needs a prompt change to behave, that is a finding: you are choosing between prompt-and-model pairs, and each pair gets its own run. Step 5: compare behavior per example, and repeat the run Compare each candidate with the baseline example by example: which tools it called, with which arguments, in what order, and whether the final answer says the same thing. One average score hides the handful of examples where the agent called the wrong tool. The guide to evaluating agent tool calls https://www.evalshift.dev/blog/evaluating-agent-tool-calls covers each check. Then run it more than once. A single run gives you the single-attempt score from reason 2. Three to five samples per example show which examples flip between runs, and those are the ones your users will hit. Step 6: re-run when the model, the prompt or the tools change The result describes one model version in one harness. Re-run the suite when the provider ships a new version, when you edit the prompt or a tool description, and when a model you depend on is scheduled for retirement https://www.evalshift.dev/blog/llm-model-deprecation-schedule . A new release that tops a leaderboard is a reason to add it to the shortlist and run step 4. Doing this with EvalShift This is the comparison EvalShift runs. Its capture SDK records real calls and promotes them into a frozen golden suite, and every tag on an example becomes a slice, so tagging by route gives per-route results. evalshift compare --from