{"slug": "llm-benchmarks-why-the-leaderboard-doesn-t-predict-your-agent", "title": "LLM benchmarks: why the leaderboard doesn't predict your agent", "summary": "Six major LLM leaderboards reviewed side by side on November 10, 2026 named four different models as best, with LMArena's text board ranking Gemini 4 Carbon first, the Artificial Analysis Intelligence Index and LiveBench ranking Claude Fable 5.2 first, Terminal-Bench and LMArena's WebDev board ranking Claude Opus 5.5 first, and Sierra's τ³-bench banking board ranking Qwen 3.8 Max first, according to EvalShift developer and post author. The post attributes the disagreement to each score measuring a model inside one harness, on one task set, under one grader, citing GPT-5 scoring 49.6% in one agent harness versus 33.9% in another on Terminal-Bench 2.0, a 6-point swing Anthropic measured from container resources alone, and roughly half of AI patches passing SWE-bench Verified's tests that project maintainers would not merge. The author argues leaderboards are useful only for shortlisting two or three models, not for predicting which one an agent should run.", "body_md": "[← all posts](https://www.evalshift.dev/blog)\n\n# LLM benchmarks: why the leaderboard doesn't predict your agent\n\nWhat LLM benchmarks measure, where the November 2026 leaderboards disagree on the best model, and how to test a shortlist on your own agent's traffic.\n\nAn LLM benchmark is a fixed set of tasks with a scoring rule, and an LLM leaderboard is a table that ranks models by it. On November 10, 2026 I read the main ones side by side. LMArena's text board had Gemini 4 Carbon on top. The Artificial Analysis Intelligence Index and LiveBench had Claude Fable 5.2. Terminal-Bench and LMArena's WebDev board had Claude Opus 5.5. Sierra's τ³-bench banking board, the one built around a customer-service agent, had Qwen 3.8 Max. Six boards named four different models as the best one.\n\nThey disagree because each score measures a model inside one harness, on one set of tasks, under one grader. Your agent shares only the model. On Terminal-Bench 2.0 the same model, GPT-5, scored 49.6% in one agent harness and 33.9% in another. On τ-bench, a model that completes a task once often fails it when asked again. Anthropic measured a 6-point swing on Terminal-Bench 2.0 from container resources alone. And roughly half of the AI patches that pass SWE-bench Verified's tests would not be merged by the projects' own maintainers.\n\nA leaderboard is still the right place to start, because it tells you which two or three models are worth testing. It can't tell you which of them your agent should run. The rest of this post covers what the main benchmarks measure, what the boards say about the best LLM for coding and for agents this month, five reasons the scores don't carry over, and how to test a shortlist on your own traffic. I build EvalShift, which automates that last part, but the method works with any tooling.\n\n## What an LLM benchmark is\n\nEvery LLM benchmark has three parts: a set of **tasks**, a **harness** that puts each task in\nfront of the model (the prompt, the tools, the loop, the time and step limits), and a **grader**\nthat turns the output into a score. A leaderboard publishes the scores as a ranking. Two\nbenchmarks with the same name on the tin, \"coding\" or \"agents\", can differ in all three.\n\n| Benchmark | Tasks | Graded by | Harness | \n|---|---|---|---|\n| MMLU, GPQA Diamond | Multiple-choice questions (GPQA Diamond has 198) | Answer key | One prompt per question | \n| [LMArena ↗](https://arena.ai/how-it-works) | Whatever its users type | A human vote between two anonymous models | A chat window | \n| [SWE-bench Verified ↗](https://www.swebench.com/verified.html) | 500 real GitHub issues | The repository's tests | Any scaffold on the main board; a \"bash only\" board fixes one | \n| [SWE-bench Pro ↗](https://labs.scale.com/leaderboard/swe_bench_pro_public) | 1,865 tasks from 41 repositories, 731 of them public | Tests | Varies by entry | \n| [Terminal-Bench ↗](https://www.tbench.ai/leaderboard) | Containerized terminal tasks | A verifier script | Each row is a model paired with an agent | \n| [Aider polyglot ↗](https://aider.chat/docs/leaderboards/) | 225 Exercism exercises in six languages | Tests | Aider | \n| [τ-bench ↗](https://taubench.com/) | Customer-service conversations with a simulated user, tools and a policy | The final database state, over repeated trials | Sierra's scaffold | \n| [GAIA ↗](https://huggingface.co/spaces/gaia-benchmark/leaderboard) | More than 450 questions with one right answer | Exact answer | Whole agent systems are submitted | \n| [OSWorld-Verified ↗](https://os-world.github.io/) | 369 desktop tasks | Execution-based scripts | Run by the OSWorld team | \n| [Artificial Analysis Intelligence Index ↗](https://artificialanalysis.ai/methodology/intelligence-benchmarking) | Nine evals in four equally weighted categories | Mixed | Artificial Analysis runs everything itself | \n| [LiveBench ↗](https://livebench.ai/#/methodology) | Seven suites, five of them agentic | Scripts and rubrics | One run per model | \n| [BenchLM ↗](https://benchlm.ai/methodology) | None of its own | Other people's benchmarks, calibrated onto one scale | None | \n\nThree things to notice before reading any ranking:\n\n- +**The oldest kind is saturated.** PhD-level experts score about 65% on GPQA Diamond. The top\nentry on[Artificial Analysis's GPQA page ↗](https://artificialanalysis.ai/evaluations/gpqa-diamond) is\n96.3%, and the next two are at 96.1% and 95.3%. A benchmark where the frontier models all\nscore in the mid-90s no longer separates them.\n- +**\"Agentic\" boards rank systems, not models.** A Terminal-Bench row is a model plus an agent.\nGAIA's top three entries this month are all multi-model harnesses. The score belongs to the\npair.\n- +**Composite indexes are weighted averages of the above.** LiveBench's overall score counts its\nlong agentic suite twice and its coding and design suites one and a half times. BenchLM runs\nno evals at all and labels its top two rows \"Estimated\". A composite inherits every limit of\nits parts and adds a choice of weights.\n\n## The LLM leaderboard in November 2026: the best model depends on the board\n\nHere is the top entry on each board as I read it on November 10, 2026. Scores are as printed, with the board's own error bars where it shows them.\n\n| Leaderboard | Top entry | Score | Runner-up | \n|---|---|---|---|\n| LMArena, text | Gemini 4 Carbon (High), marked Preliminary | 1526 ±10 | Gemini 4 Argon (High), 1522 ±5 | \n| Artificial Analysis Intelligence Index v5.0.1 | Claude Fable 5.2 (max) | 65 | Gemini 4 Carbon (high), 60, after two more Fable 5.2 settings at 62 | \n| LiveBench, overall | Claude Fable 5.2 (xhigh) | 84.9 | GPT-6 Astra, 84.5 | \n| LiveBench, coding | Claude Fable 5.1 | 89.7 | GPT-6.1 Terra, 89.2 | \n| LMArena, WebDev | Claude Opus 5.5 (Max) | 1818 ±12 | Claude Sonnet 5.5 (xHigh), 1774 ±10 | \n| Terminal-Bench 4.1 | Claude Opus 5.5 (max) + Claude Code | 65.8% ±3.1 | Claude Sonnet 5.5 (max) + Claude Code, 63.9% ±3.0 | \n| LiveBench, agentic | Claude Fable 5.2 | 82.9 | GPT-6 Astra, 75.6 | \n| OSWorld-Verified | Claude Fable 5 | 86.0% | Claude Opus 5, 83.4% | \n| τ³-bench, banking | Qwen 3.8 Max | 55.2% | Claude Opus 5, 48.7% | \n| GAIA, test set | openJiuwen-deepagent and Keen AgenticOS V1.1, tied | 94.35 | Shadow Harness V8.01, 94.02 | \n\nSources: [LMArena text ↗](https://arena.ai/leaderboard/text) and\n[WebDev ↗](https://arena.ai/leaderboard/webdev),\n[Artificial Analysis ↗](https://artificialanalysis.ai/leaderboards/models),\n[LiveBench ↗](https://livebench.ai/), [Terminal-Bench ↗](https://www.tbench.ai/leaderboard),\n[OSWorld ↗](https://os-world.github.io/), [τ-bench ↗](https://taubench.com/) and\n[GAIA ↗](https://huggingface.co/spaces/gaia-benchmark/leaderboard).\n\n### Best LLM for coding\n\nBy these boards, the best LLM for coding in November 2026 is Claude Opus 5.5, Claude Fable 5.1 or\nClaude Fable 5.2, depending on which one you open. Opus 5.5 leads Terminal-Bench 4.1 and LMArena\nWebDev. Fable 5.1 leads LiveBench's coding column at 89.7, where Fable 5.2, the model that leads\nLiveBench overall, scores 79.2. BenchLM's\n[coding ranking ↗](https://benchlm.ai/coding) puts Fable 5.2 first.\n\nThe best-known coding leaderboards can't settle it, because none of them lists those three models. SWE-bench Verified's newest entry is dated February 26, 2026, and its top row is Claude 4.5 Opus with a third-party agent at 79.2%. SWE-bench Pro's public board is led by Muse Spark 1.1 at 61.5%. Aider's polyglot board was last updated in November 2025. A leaderboard only ranks the models someone has run on it.\n\n### Best LLM for agents\n\nFor agents the boards agree less, because \"agent\" means a different job on each. LiveBench's\nagentic suite and BenchLM's [agentic ranking ↗](https://benchlm.ai/agentic) put Claude Fable 5.2\nfirst. OSWorld-Verified, which is desktop computer use, has Claude Fable 5. τ³-bench banking,\nwhere the agent talks to a customer, looks up policy and calls tools, has Qwen 3.8 Max 6.5\npoints ahead of the next model. GAIA's top entries are harnesses that combine several models, so\nits ranking says little about any one of them.\n\nTwo more things in that table matter as much as the names:\n\n- +**Most gaps at the top are inside the error bars.** LMArena's text leader is 4 points ahead\nwith a ±10 interval. Terminal-Bench's top two are 1.9 points apart with ±3 on each. LiveBench's\ntop three are within 0.9 points of each other, and its methodology page says each score comes\nfrom a single run.\n- +**The same model changes rank between columns of one board.** Fable 5.2 is first on LiveBench\noverall and more than 10 points behind the leader on its coding suite. \"Best overall\" is an\naverage, and your agent doesn't run the average.\n\n## Why an LLM benchmark score doesn't carry over to your agent\n\nFive reasons, each with published numbers.\n\n### 1. The score belongs to a model and a harness, and you run a different harness\n\nThe [Terminal-Bench 2.0 paper ↗](https://arxiv.org/abs/2601.11868) ran the same models inside\nseveral agent harnesses, at least five times each:\n\n| Model | Best harness | Worst harness | Spread | \n|---|---|---|---|\n| GPT-5 | Codex CLI, 49.6% | Mini-SWE-Agent, 33.9% | 15.7 points | \n| Gemini 2.5 Pro | Terminus 2, 32.6% | OpenHands, 15.7% | 16.9 points | \n| Claude Opus 4.5 | Terminus 2, 57.8% | OpenHands, 51.9% | 5.9 points | \n| Claude Sonnet 4.5 | Terminus 2, 42.8% | Claude Code, 40.1% | 2.7 points | \n\nWhich model wins depends on the harness. Under Codex CLI, GPT-5 beats every Claude Sonnet 4.5 row. Under Terminus 2, Sonnet 4.5 beats GPT-5 by 7.6 points. The paper's own reading is that the model usually matters more than the scaffold, and across the whole table that holds. For one model in one harness, it can be the smaller effect.\n\nThe harness can be tuned with the model held fixed.\n[LangChain ↗](https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering) took\nits coding agent from 52.8 to 66.5 on Terminal-Bench 2.0 by changing only the harness around\n`gpt-5.2-codex`. The Princeton-led team behind the\n[Holistic Agent Leaderboard ↗](https://arxiv.org/abs/2510.11977) ran 21,730 agent rollouts across\n9 models and 9 benchmarks, found that scaffolds change both accuracy and cost, and found that\nhigher reasoning effort did not improve accuracy in 21 of 36 runs.\n\nYour harness is your system prompt, your tool definitions and your loop, and no leaderboard has\nrun it. Small parts of it move results. IBM researchers\n[rephrased the user requests ↗](https://arxiv.org/abs/2504.00914) in a function-calling benchmark\nwithout changing their meaning: `gpt-4o-mini` fell from 0.925 to 0.765 on tool-call accuracy and\nClaude 3.5 Sonnet from 0.915 to 0.845. Anthropic's\n[guide to writing tools for agents ↗](https://www.anthropic.com/engineering/writing-tools-for-agents)\nreports that small edits to tool descriptions produced large changes in its tool-use evals.\n\n### 2. One success is not reliability\n\nMost leaderboards report whether the model completed each task once. A production agent meets\nthe same kind of request thousands of times.\n[τ-bench ↗](https://arxiv.org/abs/2406.12045) introduced pass^k for this: the chance that all *k*\nindependent attempts at a task succeed. In the original 2024 paper, a `gpt-4o` function-calling\nagent completed 61.2% of retail tasks on one attempt and under 25% on all of eight.\n\nNewer models have the same gap from a higher starting point. Sierra's\n[leaderboard submissions ↗](https://github.com/sierra-research/tau2-bench/tree/main/web/leaderboard/public/submissions)\nreport four trials per task:\n\n| Model, domain | 1 of 1 | 4 of 4 | \n|---|---|---|\n| Claude Opus 4.5, retail | 79.6% | 51.8% | \n| Claude Opus 4.5, airline | 84.0% | 70.0% | \n| Claude Opus 4.5, telecom | 92.3% | 78.1% | \n\nThe arithmetic is unforgiving: an agent that succeeds 90% of the time on a task succeeds five\ntimes in a row 59% of the time. A 2026 Princeton paper,\n[Towards a Science of AI Agent Reliability ↗](https://arxiv.org/abs/2602.16666), ran 15 models\nfive times each on GAIA and τ-bench and found that reliability has improved more slowly than\nheadline accuracy. A leaderboard column of single-attempt scores can't show you that.\n\n### 3. The gap between two models is often smaller than the noise\n\nAnthropic ran Terminal-Bench 2.0 under different container resource limits and found a\n[6-point gap ↗](https://www.anthropic.com/engineering/infrastructure-noise) between the most and\nleast generous setups, with the model unchanged. Its rule of thumb is to distrust leaderboard\ndifferences under 3 points until the configurations are documented and matched. A separate study\nof 60,000 agent trajectories on SWE-bench Verified\n([On Randomness in Agentic Evals ↗](https://arxiv.org/abs/2602.07150)) found that a single-run\nscore moves by 2.2 to 6.0 points depending on which run you happened to get, and that the spread\npersists at temperature 0.\n\nFew benchmarks report this. An Oxford-led\n[review of 445 LLM benchmarks ↗](https://arxiv.org/abs/2511.04703) found that 16% ran any\nstatistical test. Apply a 3-point rule to the November table and first place on LiveBench\noverall, LiveBench coding, Terminal-Bench, OSWorld-Verified and GAIA becomes a tie.\n\n### 4. Public tasks leak, and graders have holes\n\nA public benchmark's tasks and answers end up in training data, and its grader is only as strict as the tests somebody wrote.\n\n- +**Contamination.** On February 23, 2026 OpenAI[stopped reporting SWE-bench Verified ↗](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/) and asked other labs to do the same. It audited 138 problems its model did not solve\nconsistently and found material flaws in the tests or the task description in 59.4% of them. It also found\nthat frontier models from three labs could reproduce the original fixes or task details from\nmemory.\n- +**Private code is harder.**[SWE-bench Pro ↗](https://arxiv.org/abs/2509.16941v1) has a public\nset and a commercial set drawn from private startup codebases. In its first release GPT-5\nscored 23.3% on the public set and 14.9% on the commercial one, and Claude Opus 4.1 went from\n22.7% to 17.8%. The model that was ahead on the public set was behind on the private one.\n- +**Passing the tests is not the same as doing the job.** METR asked four maintainers of three\nSWE-bench repositories to review 296 AI-written pull requests and[estimated ↗](https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/) that roughly half of the ones that pass the benchmark's tests would not be merged. METR notes\nthe agents had no chance to respond to review, so this is a gap between the grader and the\njob, not a ceiling on the models.\n- +**Harnesses can be exploited.** In April 2026 a UC Berkeley group[published proof-of-concept exploits ↗](https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/) that scored 100% on SWE-bench Verified, SWE-bench Pro and Terminal-Bench, about 98% on GAIA\nand 73% on OSWorld without solving a task. That shows what the graders would accept. It is not\nevidence that any leaderboard entry cheated.\n- +**Arena scores can be shopped.**[The Leaderboard Illusion ↗](https://arxiv.org/abs/2504.20879) documented 27 private model\nvariants tested by one lab on Chatbot Arena before a release, and argued that a lab can\npublish only the best-scoring one.[LMArena's response ↗](https://news.lmarena.ai/our-response) disputes the size of the effect and puts it at around 11 Elo points.\n\n### 5. Your tasks are not the benchmark's tasks\n\nA benchmark score is an average over somebody else's task distribution. Your agent has a handful of routes, its own tools, its own policies and its own users. No public board averages over those.\n\nThe model providers say this themselves.\n[Anthropic's ↗](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) and\n[OpenAI's ↗](https://developers.openai.com/api/docs/guides/evaluation-best-practices) evaluation\nguides both tell teams to build evals that mirror their real task distribution, and\n[Google's ↗](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview)\nsays its evaluation service gives insight that public leaderboards and general benchmarks\ncan't. Databricks\n[benchmarked coding agents on its own codebase ↗](https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase)\nin July 2026 because the public benchmarks couldn't answer its questions, and found an\nopen-source model statistically tied with the top scorer at two thirds of its cost per task.\n\nI have seen both outcomes on real agents. I replayed 16 real examples from one agent on its\ncurrent model and a cheaper candidate: 58.8% cheaper, 75.2% faster, no failed calls, and the\ncandidate picked a different tool on 25% of comparisons against a 10% limit. The report came\nback [FAIL](https://www.evalshift.dev/blog/cheaper-faster-and-still-a-fail). A customer-support agent with six tools\nmoved to a cheaper model over 120 production examples and\n[held every budget](https://www.evalshift.dev/blog/what-a-passing-migration-proves) at 71.2% lower cost. Nothing on a\nleaderboard told me which of the two would pass.\n\n## What LLM leaderboards are good for\n\nNone of this makes leaderboards useless. They do three jobs well:\n\n- +**Shortlisting.** A model 20 points behind on every board that resembles your task is\nunlikely to win on yours. Leaderboards cut a field of dozens to two or three candidates.\n- +**Rough tiers.** A gap much larger than the noise is real information, even when a 2-point\ngap is not.\n- +**Price and speed.** Artificial Analysis measures cost and latency alongside quality, which\ntells you what a candidate would have to beat to be worth a switch. The[cheapest LLM API comparison](https://www.evalshift.dev/blog/cheapest-llm-api) covers why list price is still not what\nan agent costs.\n\nTo read one, ask five questions of the board before you look at the names:\n\n1. +**What are the tasks?** Find the board whose tasks look most like your agent's. A\ncustomer-service agent is closer to τ-bench than to Terminal-Bench.\n2. +**What harness produced the score?** A row that pairs the model with its vendor's own agent\ntells you about that agent. Look for a board, or a column, that holds the harness fixed.\n3. +**Who graded it?** Tests, an answer key, a human vote and an LLM judge fail in different\nways.\n4. +**How many runs, and is there an error bar?** One run with no interval can't separate\nneighbors.\n5. +**How old are the tasks, and are they public?** Public tasks older than the model's training\ncutoff may be in its training data. Check the date of the newest entry too.\n\n## How to pick a model for your agent\n\nSix steps. They are the same whether you are choosing a first model or deciding whether a new release should replace the one you run.\n\n## Step 1: shortlist from the boards closest to your task\n\nPick two or three candidates from the leaderboards whose tasks and harness resemble yours, plus the model you run today as the baseline. Include one model cheaper than you think you need: the passing report above was a 71.2% saving that no ranking would have suggested.\n\n## Step 2: build the test from your own traffic\n\nTake 100 to 300 real inputs from production, stratified by route or intent so the rare and\nexpensive paths are in the sample. For an agent, take whole conversations with the tool results\nthey produced. A\n[golden suite built from production traffic](https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic)\nis the reusable version, and it is a benchmark nobody has trained on.\n\nSize the routes you care about, not only the total. Zero failures in 20 examples still allows a\ntrue failure rate of about 15% at 95% confidence, and zero in 100 brings that to about 3%.\n[How many eval cases you need](https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need) has the full sizing.\n\n## Step 3: write the limits before the run\n\nFor each route, write down the most regressions you will accept, the most tool-selection divergence, and the cost or latency change that would make a switch worth it. Routes that move money or touch security get zero. Limits written after the numbers are in tend to match the numbers.\n\n## Step 4: run every candidate inside your own harness\n\nRun each candidate with your system prompt, your tool definitions and your loop, on the same inputs as the baseline. This is the step a leaderboard can't do for you, and reason 1 above is why it matters. If a candidate needs a prompt change to behave, that is a finding: you are choosing between prompt-and-model pairs, and each pair gets its own run.\n\n## Step 5: compare behavior per example, and repeat the run\n\nCompare each candidate with the baseline example by example: which tools it called, with which\narguments, in what order, and whether the final answer says the same thing. One average score\nhides the handful of examples where the agent called the wrong tool. The\n[guide to evaluating agent tool calls](https://www.evalshift.dev/blog/evaluating-agent-tool-calls) covers each check.\n\nThen run it more than once. A single run gives you the single-attempt score from reason 2. Three to five samples per example show which examples flip between runs, and those are the ones your users will hit.\n\n## Step 6: re-run when the model, the prompt or the tools change\n\nThe result describes one model version in one harness. Re-run the suite when the provider ships\na new version, when you edit the prompt or a tool description, and when a model you depend on is\n[scheduled for retirement](https://www.evalshift.dev/blog/llm-model-deprecation-schedule). A new release that tops a\nleaderboard is a reason to add it to the shortlist and run step 4.\n\n## Doing this with EvalShift\n\nThis is the comparison EvalShift runs. Its capture SDK records real calls and promotes them into\na frozen golden suite, and every tag on an example becomes a slice, so tagging by route gives\nper-route results. `evalshift compare --from <current-model> --to <candidate>` replays the suite\non both models, scores every pair with the evaluators you configure (tool calls and arguments,\nstructure, semantic similarity, an LLM judge), and measures cost and latency from the run's own\ncalls. For a shortlist, run it once per candidate against the same baseline.\n\nThe limits from step 3 go in `migration_policy` in `evalshift.yaml`, with per-route overrides\nunder `slices`, and repeated runs are one setting:\n\n```\ndefaults:\n  samples_per_example: 3\n\nmigration_policy:\n  max_overall_regression_rate: 0.05\n  max_tool_divergence: 0.05\n  slices:\n    refund:\n      max_overall_regression_rate: 0.0\n      max_tool_divergence: 0.0\n      max_tool_argument_drift: 0.0\n```\n\nThe report gives a verdict, the per-slice results and the per-example diffs. The\n[step-by-step migration test guide](https://www.evalshift.dev/blog/test-llm-model-migration-before-you-ship) shows the\nwhole flow, and the [methodology docs](https://www.evalshift.dev/docs/methodology) explain the paired statistics behind\nthe verdict.\n\nAn LLM leaderboard tells you which models are worth an afternoon of testing. Which one your agent should run is a measurement you have to take yourself, on your own traffic, inside your own harness.\n\n## FAQ\n\n**What is an LLM benchmark?** An LLM benchmark is a fixed set of tasks, a harness that presents\neach task to the model, and a grader that turns the output into a score. Examples are SWE-bench\nVerified (500 GitHub issues graded by tests), τ-bench (customer-service conversations graded by\nthe final database state) and LMArena (human votes between two anonymous models).\n\n**What is an LLM leaderboard?** An LLM leaderboard is a ranking of models by their score on one\nbenchmark or on a weighted mix of several. Some run every model themselves (Artificial Analysis,\nLiveBench), some accept submissions (SWE-bench, GAIA, Terminal-Bench), and some aggregate other\npeople's results (BenchLM).\n\n**What is the best LLM for coding?** The boards disagree. As read on November 10, 2026, Claude\nOpus 5.5 leads Terminal-Bench 4.1 and LMArena WebDev, Claude Fable 5.1 leads LiveBench's coding\nsuite, and BenchLM's coding ranking puts Claude Fable 5.2 first. SWE-bench Verified, SWE-bench\nPro and Aider's polyglot board list none of those three models. Shortlist from the boards that\ndo and test on your own repository.\n\n**What is the best LLM for agents?** It depends on the agent's job. In November 2026 Claude\nFable 5.2 leads LiveBench's agentic suite, Claude Fable 5 leads OSWorld-Verified for desktop\ncomputer use, and Qwen 3.8 Max leads τ³-bench banking for customer-service tool use. GAIA's top\nentries are multi-model harnesses. Use the board closest to your task to shortlist, then replay\nyour own traffic.\n\n**Are LLM benchmarks reliable?** They are reliable for large gaps and unreliable for small ones.\nAnthropic measured a 6-point swing on Terminal-Bench 2.0 from container resources alone, single\nruns on SWE-bench Verified vary by 2.2 to 6.0 points, and an Oxford-led review of 445 benchmarks\nfound that 16% ran any statistical test. Public tasks also leak into training data, one of the\nreasons OpenAI stopped reporting SWE-bench Verified in February 2026.\n\n**Why do LLM leaderboards disagree?** Each one measures a different combination of tasks, harness\nand grader. The same model scores differently in different agent harnesses (GPT-5 ranged from\n33.9% to 49.6% on Terminal-Bench 2.0), composite indexes weight their parts differently, and\nmany gaps at the top are smaller than the boards' own error bars.\n\n**How do I benchmark an LLM for my own application?** Replay 100 to 300 real production inputs\non each shortlisted model, inside your own prompt and tools, with limits written before the run.\nCompare each candidate with your current model example by example, including tool calls and\narguments, and repeat the run to see which examples flip.\n\n## Keep reading\n\n- +[How to evaluate AI agents](https://www.evalshift.dev/blog/how-to-evaluate-ai-agents) — what to check in an agent's\noutput and tool calls once you have your own test set.\n- +[The cheapest LLM API in 2026, and whether it is good enough for your agent](https://www.evalshift.dev/blog/cheapest-llm-api) — the price side of the same decision.\n- +[LLM routing cuts costs. Here's how to check it doesn't cut quality](https://www.evalshift.dev/blog/llm-router) — why\nrouter benchmarks have the same problem as model benchmarks.", "url": "https://wpnews.pro/news/llm-benchmarks-why-the-leaderboard-doesn-t-predict-your-agent", "canonical_source": "https://www.evalshift.dev/blog/llm-benchmarks", "published_at": "2026-11-10 21:52:39.974268+00:00", "updated_at": "2026-11-10 21:52:42.350316+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-research", "ai-tools"], "entities": ["EvalShift", "LMArena", "Gemini 4 Carbon", "Artificial Analysis Intelligence Index", "LiveBench", "Claude Fable 5.2", "Claude Opus 5.5", "Terminal-Bench"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llm-benchmarks-why-the-leaderboard-doesn-t-predict-your-agent", "markdown": "https://wpnews.pro/news/llm-benchmarks-why-the-leaderboard-doesn-t-predict-your-agent.md", "text": "https://wpnews.pro/news/llm-benchmarks-why-the-leaderboard-doesn-t-predict-your-agent.txt", "jsonld": "https://wpnews.pro/news/llm-benchmarks-why-the-leaderboard-doesn-t-predict-your-agent.jsonld"}}