What LLM benchmarks measure, where the November 2026 leaderboards disagree on the best model, and how to test a shortlist on your own agent's traffic.
An LLM benchmark is a fixed set of tasks with a scoring rule, and an LLM leaderboard is a table that ranks models by it. On November 10, 2026 I read the main ones side by side. LMArena's text board had Gemini 4 Carbon on top. The Artificial Analysis Intelligence Index and LiveBench had Claude Fable 5.2. Terminal-Bench and LMArena's WebDev board had Claude Opus 5.5. Sierra's τ³-bench banking board, the one built around a customer-service agent, had Qwen 3.8 Max. Six boards named four different models as the best one.
They disagree because each score measures a model inside one harness, on one set of tasks, under one grader. Your agent shares only the model. On Terminal-Bench 2.0 the same model, GPT-5, scored 49.6% in one agent harness and 33.9% in another. On τ-bench, a model that completes a task once often fails it when asked again. Anthropic measured a 6-point swing on Terminal-Bench 2.0 from container resources alone. And roughly half of the AI patches that pass SWE-bench Verified's tests would not be merged by the projects' own maintainers.
A leaderboard is still the right place to start, because it tells you which two or three models are worth testing. It can't tell you which of them your agent should run. The rest of this post covers what the main benchmarks measure, what the boards say about the best LLM for coding and for agents this month, five reasons the scores don't carry over, and how to test a shortlist on your own traffic. I build EvalShift, which automates that last part, but the method works with any tooling.
What an LLM benchmark is #
Every LLM benchmark has three parts: a set of tasks, a harness that puts each task in front of the model (the prompt, the tools, the loop, the time and step limits), and a grader that turns the output into a score. A leaderboard publishes the scores as a ranking. Two benchmarks with the same name on the tin, "coding" or "agents", can differ in all three.
| Benchmark | Tasks | Graded by | Harness |
|---|---|---|---|
| MMLU, GPQA Diamond | Multiple-choice questions (GPQA Diamond has 198) | Answer key | One prompt per question |
| LMArena ↗ | Whatever its users type | A human vote between two anonymous models | A chat window |
| SWE-bench Verified ↗ | 500 real GitHub issues | The repository's tests | Any scaffold on the main board; a "bash only" board fixes one |
| SWE-bench Pro ↗ | 1,865 tasks from 41 repositories, 731 of them public | Tests | Varies by entry |
| Terminal-Bench ↗ | Containerized terminal tasks | A verifier script | Each row is a model paired with an agent |
| Aider polyglot ↗ | 225 Exercism exercises in six languages | Tests | Aider |
| τ-bench ↗ | Customer-service conversations with a simulated user, tools and a policy | The final database state, over repeated trials | Sierra's scaffold |
| GAIA ↗ | More than 450 questions with one right answer | Exact answer | Whole agent systems are submitted |
| OSWorld-Verified ↗ | 369 desktop tasks | Execution-based scripts | Run by the OSWorld team |
| Artificial Analysis Intelligence Index ↗ | Nine evals in four equally weighted categories | Mixed | Artificial Analysis runs everything itself |
| LiveBench ↗ | Seven suites, five of them agentic | Scripts and rubrics | One run per model |
| BenchLM ↗ | None of its own | Other people's benchmarks, calibrated onto one scale | None |
Three things to notice before reading any ranking:
- +The oldest kind is saturated. PhD-level experts score about 65% on GPQA Diamond. The top entry onArtificial Analysis's GPQA page ↗ is 96.3%, and the next two are at 96.1% and 95.3%. A benchmark where the frontier models all score in the mid-90s no longer separates them.
- +"Agentic" boards rank systems, not models. A Terminal-Bench row is a model plus an agent. GAIA's top three entries this month are all multi-model harnesses. The score belongs to the pair.
- +Composite indexes are weighted averages of the above. LiveBench's overall score counts its long agentic suite twice and its coding and design suites one and a half times. BenchLM runs no evals at all and labels its top two rows "Estimated". A composite inherits every limit of its parts and adds a choice of weights.
The LLM leaderboard in November 2026: the best model depends on the board #
Here is the top entry on each board as I read it on November 10, 2026. Scores are as printed, with the board's own error bars where it shows them.
| Leaderboard | Top entry | Score | Runner-up |
|---|---|---|---|
| LMArena, text | Gemini 4 Carbon (High), marked Preliminary | 1526 ±10 | Gemini 4 Argon (High), 1522 ±5 |
| Artificial Analysis Intelligence Index v5.0.1 | Claude Fable 5.2 (max) | 65 | Gemini 4 Carbon (high), 60, after two more Fable 5.2 settings at 62 |
| LiveBench, overall | Claude Fable 5.2 (xhigh) | 84.9 | GPT-6 Astra, 84.5 |
| LiveBench, coding | Claude Fable 5.1 | 89.7 | GPT-6.1 Terra, 89.2 |
| LMArena, WebDev | Claude Opus 5.5 (Max) | 1818 ±12 | Claude Sonnet 5.5 (xHigh), 1774 ±10 |
| Terminal-Bench 4.1 | Claude Opus 5.5 (max) + Claude Code | 65.8% ±3.1 | Claude Sonnet 5.5 (max) + Claude Code, 63.9% ±3.0 |
| LiveBench, agentic | Claude Fable 5.2 | 82.9 | GPT-6 Astra, 75.6 |
| OSWorld-Verified | Claude Fable 5 | 86.0% | Claude Opus 5, 83.4% |
| τ³-bench, banking | Qwen 3.8 Max | 55.2% | Claude Opus 5, 48.7% |
| GAIA, test set | openJiuwen-deepagent and Keen AgenticOS V1.1, tied | 94.35 | Shadow Harness V8.01, 94.02 |
Sources: LMArena text ↗ and WebDev ↗, Artificial Analysis ↗, LiveBench ↗, Terminal-Bench ↗, OSWorld ↗, τ-bench ↗ and GAIA ↗.
Best LLM for coding
By these boards, the best LLM for coding in November 2026 is Claude Opus 5.5, Claude Fable 5.1 or Claude Fable 5.2, depending on which one you open. Opus 5.5 leads Terminal-Bench 4.1 and LMArena WebDev. Fable 5.1 leads LiveBench's coding column at 89.7, where Fable 5.2, the model that leads LiveBench overall, scores 79.2. BenchLM's coding ranking ↗ puts Fable 5.2 first.
The best-known coding leaderboards can't settle it, because none of them lists those three models. SWE-bench Verified's newest entry is dated February 26, 2026, and its top row is Claude 4.5 Opus with a third-party agent at 79.2%. SWE-bench Pro's public board is led by Muse Spark 1.1 at 61.5%. Aider's polyglot board was last updated in November 2025. A leaderboard only ranks the models someone has run on it.
Best LLM for agents
For agents the boards agree less, because "agent" means a different job on each. LiveBench's agentic suite and BenchLM's agentic ranking ↗ put Claude Fable 5.2 first. OSWorld-Verified, which is desktop computer use, has Claude Fable 5. τ³-bench banking, where the agent talks to a customer, looks up policy and calls tools, has Qwen 3.8 Max 6.5 points ahead of the next model. GAIA's top entries are harnesses that combine several models, so its ranking says little about any one of them.
Two more things in that table matter as much as the names:
- +Most gaps at the top are inside the error bars. LMArena's text leader is 4 points ahead with a ±10 interval. Terminal-Bench's top two are 1.9 points apart with ±3 on each. LiveBench's top three are within 0.9 points of each other, and its methodology page says each score comes from a single run.
- +The same model changes rank between columns of one board. Fable 5.2 is first on LiveBench overall and more than 10 points behind the leader on its coding suite. "Best overall" is an average, and your agent doesn't run the average.
Why an LLM benchmark score doesn't carry over to your agent #
Five reasons, each with published numbers.
1. The score belongs to a model and a harness, and you run a different harness
The Terminal-Bench 2.0 paper ↗ ran the same models inside several agent harnesses, at least five times each:
| Model | Best harness | Worst harness | Spread |
|---|---|---|---|
| GPT-5 | Codex CLI, 49.6% | Mini-SWE-Agent, 33.9% | 15.7 points |
| Gemini 2.5 Pro | Terminus 2, 32.6% | OpenHands, 15.7% | 16.9 points |
| Claude Opus 4.5 | Terminus 2, 57.8% | OpenHands, 51.9% | 5.9 points |
| Claude Sonnet 4.5 | Terminus 2, 42.8% | Claude Code, 40.1% | 2.7 points |
Which model wins depends on the harness. Under Codex CLI, GPT-5 beats every Claude Sonnet 4.5 row. Under Terminus 2, Sonnet 4.5 beats GPT-5 by 7.6 points. The paper's own reading is that the model usually matters more than the scaffold, and across the whole table that holds. For one model in one harness, it can be the smaller effect.
The harness can be tuned with the model held fixed.
LangChain ↗ took
its coding agent from 52.8 to 66.5 on Terminal-Bench 2.0 by changing only the harness around
gpt-5.2-codex. The Princeton-led team behind the
Holistic Agent Leaderboard ↗ ran 21,730 agent rollouts across
9 models and 9 benchmarks, found that scaffolds change both accuracy and cost, and found that
higher reasoning effort did not improve accuracy in 21 of 36 runs.
Your harness is your system prompt, your tool definitions and your loop, and no leaderboard has
run it. Small parts of it move results. IBM researchers
rephrased the user requests ↗ in a function-calling benchmark
without changing their meaning: gpt-4o-mini fell from 0.925 to 0.765 on tool-call accuracy and
Claude 3.5 Sonnet from 0.915 to 0.845. Anthropic's
guide to writing tools for agents ↗
reports that small edits to tool descriptions produced large changes in its tool-use evals.
2. One success is not reliability
Most leaderboards report whether the model completed each task once. A production agent meets
the same kind of request thousands of times.
τ-bench ↗ introduced pass^k for this: the chance that all k
independent attempts at a task succeed. In the original 2024 paper, a gpt-4o function-calling
agent completed 61.2% of retail tasks on one attempt and under 25% on all of eight.
Newer models have the same gap from a higher starting point. Sierra's leaderboard submissions ↗ report four trials per task:
| Model, domain | 1 of 1 | 4 of 4 |
|---|---|---|
| Claude Opus 4.5, retail | 79.6% | 51.8% |
| Claude Opus 4.5, airline | 84.0% | 70.0% |
| Claude Opus 4.5, telecom | 92.3% | 78.1% |
The arithmetic is unforgiving: an agent that succeeds 90% of the time on a task succeeds five times in a row 59% of the time. A 2026 Princeton paper, Towards a Science of AI Agent Reliability ↗, ran 15 models five times each on GAIA and τ-bench and found that reliability has improved more slowly than headline accuracy. A leaderboard column of single-attempt scores can't show you that.
3. The gap between two models is often smaller than the noise
Anthropic ran Terminal-Bench 2.0 under different container resource limits and found a 6-point gap ↗ between the most and least generous setups, with the model unchanged. Its rule of thumb is to distrust leaderboard differences under 3 points until the configurations are documented and matched. A separate study of 60,000 agent trajectories on SWE-bench Verified (On Randomness in Agentic Evals ↗) found that a single-run score moves by 2.2 to 6.0 points depending on which run you happened to get, and that the spread persists at temperature 0.
Few benchmarks report this. An Oxford-led review of 445 LLM benchmarks ↗ found that 16% ran any statistical test. Apply a 3-point rule to the November table and first place on LiveBench overall, LiveBench coding, Terminal-Bench, OSWorld-Verified and GAIA becomes a tie.
4. Public tasks leak, and graders have holes
A public benchmark's tasks and answers end up in training data, and its grader is only as strict as the tests somebody wrote.
- +Contamination. On February 23, 2026 OpenAIstopped reporting SWE-bench Verified ↗ and asked other labs to do the same. It audited 138 problems its model did not solve consistently and found material flaws in the tests or the task description in 59.4% of them. It also found that frontier models from three labs could reproduce the original fixes or task details from memory.
- +Private code is harder.SWE-bench Pro ↗ has a public set and a commercial set drawn from private startup codebases. In its first release GPT-5 scored 23.3% on the public set and 14.9% on the commercial one, and Claude Opus 4.1 went from 22.7% to 17.8%. The model that was ahead on the public set was behind on the private one.
- +Passing the tests is not the same as doing the job. METR asked four maintainers of three SWE-bench repositories to review 296 AI-written pull requests andestimated ↗ that roughly half of the ones that pass the benchmark's tests would not be merged. METR notes the agents had no chance to respond to review, so this is a gap between the grader and the job, not a ceiling on the models.
- +Harnesses can be exploited. In April 2026 a UC Berkeley grouppublished proof-of-concept exploits ↗ that scored 100% on SWE-bench Verified, SWE-bench Pro and Terminal-Bench, about 98% on GAIA and 73% on OSWorld without solving a task. That shows what the graders would accept. It is not evidence that any leaderboard entry cheated.
- +Arena scores can be shopped.The Leaderboard Illusion ↗ documented 27 private model variants tested by one lab on Chatbot Arena before a release, and argued that a lab can publish only the best-scoring one.LMArena's response ↗ disputes the size of the effect and puts it at around 11 Elo points.
5. Your tasks are not the benchmark's tasks
A benchmark score is an average over somebody else's task distribution. Your agent has a handful of routes, its own tools, its own policies and its own users. No public board averages over those.
The model providers say this themselves. Anthropic's ↗ and OpenAI's ↗ evaluation guides both tell teams to build evals that mirror their real task distribution, and Google's ↗ says its evaluation service gives insight that public leaderboards and general benchmarks can't. Databricks benchmarked coding agents on its own codebase ↗ in July 2026 because the public benchmarks couldn't answer its questions, and found an open-source model statistically tied with the top scorer at two thirds of its cost per task.
I have seen both outcomes on real agents. I replayed 16 real examples from one agent on its current model and a cheaper candidate: 58.8% cheaper, 75.2% faster, no failed calls, and the candidate picked a different tool on 25% of comparisons against a 10% limit. The report came back FAIL. A customer-support agent with six tools moved to a cheaper model over 120 production examples and held every budget at 71.2% lower cost. Nothing on a leaderboard told me which of the two would pass.
What LLM leaderboards are good for #
None of this makes leaderboards useless. They do three jobs well:
- +Shortlisting. A model 20 points behind on every board that resembles your task is unlikely to win on yours. Leaderboards cut a field of dozens to two or three candidates.
- +Rough tiers. A gap much larger than the noise is real information, even when a 2-point gap is not.
- +Price and speed. Artificial Analysis measures cost and latency alongside quality, which tells you what a candidate would have to beat to be worth a switch. Thecheapest LLM API comparison covers why list price is still not what an agent costs.
To read one, ask five questions of the board before you look at the names:
- +What are the tasks? Find the board whose tasks look most like your agent's. A customer-service agent is closer to τ-bench than to Terminal-Bench.
- +What harness produced the score? A row that pairs the model with its vendor's own agent tells you about that agent. Look for a board, or a column, that holds the harness fixed.
- +Who graded it? Tests, an answer key, a human vote and an LLM judge fail in different ways.
- +How many runs, and is there an error bar? One run with no interval can't separate neighbors.
- +How old are the tasks, and are they public? Public tasks older than the model's training cutoff may be in its training data. Check the date of the newest entry too.
How to pick a model for your agent #
Six steps. They are the same whether you are choosing a first model or deciding whether a new release should replace the one you run.
Step 1: shortlist from the boards closest to your task #
Pick two or three candidates from the leaderboards whose tasks and harness resemble yours, plus the model you run today as the baseline. Include one model cheaper than you think you need: the passing report above was a 71.2% saving that no ranking would have suggested.
Step 2: build the test from your own traffic #
Take 100 to 300 real inputs from production, stratified by route or intent so the rare and expensive paths are in the sample. For an agent, take whole conversations with the tool results they produced. A golden suite built from production traffic is the reusable version, and it is a benchmark nobody has trained on.
Size the routes you care about, not only the total. Zero failures in 20 examples still allows a true failure rate of about 15% at 95% confidence, and zero in 100 brings that to about 3%. How many eval cases you need has the full sizing.
Step 3: write the limits before the run #
For each route, write down the most regressions you will accept, the most tool-selection divergence, and the cost or latency change that would make a switch worth it. Routes that move money or touch security get zero. Limits written after the numbers are in tend to match the numbers.
Step 4: run every candidate inside your own harness #
Run each candidate with your system prompt, your tool definitions and your loop, on the same inputs as the baseline. This is the step a leaderboard can't do for you, and reason 1 above is why it matters. If a candidate needs a prompt change to behave, that is a finding: you are choosing between prompt-and-model pairs, and each pair gets its own run.
Step 5: compare behavior per example, and repeat the run #
Compare each candidate with the baseline example by example: which tools it called, with which arguments, in what order, and whether the final answer says the same thing. One average score hides the handful of examples where the agent called the wrong tool. The guide to evaluating agent tool calls covers each check.
Then run it more than once. A single run gives you the single-attempt score from reason 2. Three to five samples per example show which examples flip between runs, and those are the ones your users will hit.
Step 6: re-run when the model, the prompt or the tools change #
The result describes one model version in one harness. Re-run the suite when the provider ships a new version, when you edit the prompt or a tool description, and when a model you depend on is scheduled for retirement. A new release that tops a leaderboard is a reason to add it to the shortlist and run step 4.
Doing this with EvalShift #
This is the comparison EvalShift runs. Its capture SDK records real calls and promotes them into
a frozen golden suite, and every tag on an example becomes a slice, so tagging by route gives
per-route results. evalshift compare --from <current-model> --to <candidate> replays the suite
on both models, scores every pair with the evaluators you configure (tool calls and arguments,
structure, semantic similarity, an LLM judge), and measures cost and latency from the run's own
calls. For a shortlist, run it once per candidate against the same baseline.
The limits from step 3 go in migration_policy in evalshift.yaml, with per-route overrides
under slices, and repeated runs are one setting:
defaults:
samples_per_example: 3
migration_policy:
max_overall_regression_rate: 0.05
max_tool_divergence: 0.05
slices:
refund:
max_overall_regression_rate: 0.0
max_tool_divergence: 0.0
max_tool_argument_drift: 0.0
The report gives a verdict, the per-slice results and the per-example diffs. The step-by-step migration test guide shows the whole flow, and the methodology docs explain the paired statistics behind the verdict.
An LLM leaderboard tells you which models are worth an afternoon of testing. Which one your agent should run is a measurement you have to take yourself, on your own traffic, inside your own harness.
FAQ #
What is an LLM benchmark? An LLM benchmark is a fixed set of tasks, a harness that presents each task to the model, and a grader that turns the output into a score. Examples are SWE-bench Verified (500 GitHub issues graded by tests), τ-bench (customer-service conversations graded by the final database state) and LMArena (human votes between two anonymous models).
What is an LLM leaderboard? An LLM leaderboard is a ranking of models by their score on one benchmark or on a weighted mix of several. Some run every model themselves (Artificial Analysis, LiveBench), some accept submissions (SWE-bench, GAIA, Terminal-Bench), and some aggregate other people's results (BenchLM).
What is the best LLM for coding? The boards disagree. As read on November 10, 2026, Claude Opus 5.5 leads Terminal-Bench 4.1 and LMArena WebDev, Claude Fable 5.1 leads LiveBench's coding suite, and BenchLM's coding ranking puts Claude Fable 5.2 first. SWE-bench Verified, SWE-bench Pro and Aider's polyglot board list none of those three models. Shortlist from the boards that do and test on your own repository.
What is the best LLM for agents? It depends on the agent's job. In November 2026 Claude Fable 5.2 leads LiveBench's agentic suite, Claude Fable 5 leads OSWorld-Verified for desktop computer use, and Qwen 3.8 Max leads τ³-bench banking for customer-service tool use. GAIA's top entries are multi-model harnesses. Use the board closest to your task to shortlist, then replay your own traffic.
Are LLM benchmarks reliable? They are reliable for large gaps and unreliable for small ones. Anthropic measured a 6-point swing on Terminal-Bench 2.0 from container resources alone, single runs on SWE-bench Verified vary by 2.2 to 6.0 points, and an Oxford-led review of 445 benchmarks found that 16% ran any statistical test. Public tasks also leak into training data, one of the reasons OpenAI stopped reporting SWE-bench Verified in February 2026.
Why do LLM leaderboards disagree? Each one measures a different combination of tasks, harness and grader. The same model scores differently in different agent harnesses (GPT-5 ranged from 33.9% to 49.6% on Terminal-Bench 2.0), composite indexes weight their parts differently, and many gaps at the top are smaller than the boards' own error bars.
How do I benchmark an LLM for my own application? Replay 100 to 300 real production inputs on each shortlisted model, inside your own prompt and tools, with limits written before the run. Compare each candidate with your current model example by example, including tool calls and arguments, and repeat the run to see which examples flip.
Keep reading #
- +How to evaluate AI agents — what to check in an agent's output and tool calls once you have your own test set.
- +The cheapest LLM API in 2026, and whether it is good enough for your agent — the price side of the same decision.
- +LLM routing cuts costs. Here's how to check it doesn't cut quality — why router benchmarks have the same problem as model benchmarks.