# My eval said a perfect MCP server was broken. It was the eval that was lying.

> Source: <https://tengli.dev/posts/calibrating-mcp-evals.html>
> Published: 2026-07-27 00:00:00+00:00

# My eval said a perfect MCP server was broken. It was the eval that was lying.

When I added an LLM-powered eval to [mcpgrade](https://github.com/TengByte/mcpgrade), the first real run produced a result that looked like a scoop: context7 — a server with a *perfect* static score — failed tool selection 62% of the time. A model shown its two-tool catalog picked the "wrong" tool on 5 of 8 tasks.

If I had shipped that number, it would have been wrong. Not slightly wrong — systematically, unfairly wrong. This post is about how I caught it, because the failure mode generalizes to most agent benchmarks people are building right now.

## The setup

mcpgrade's `--eval`

mode works like this: it reads a server's tool catalog, synthesizes realistic single-step tasks ("find the Slack channel where the incident was discussed"), shows a model the full catalog, and measures three things — does it pick the right tool, does it fill valid arguments, and does it correctly *refuse* tasks that no tool can handle.

Round 1, on three real servers, cost about twelve cents and produced this:

| Server | Static score | Tool selection | Args | Refusal |
|---|---|---|---|---|
| context7 (2 tools) | 100 | 38% |
100% | 100% |
| server-memory (9 tools) | 81 | 93% | 100% | 100% |
| server-slack (8 tools) | 97 | 54% |
100% | 100% |

Two servers with excellent static scores, apparently failing live. Either static analysis was worthless, or the eval was broken.

## The eval was broken

Every "miss" traced to one cause. Slack's `post_message`

needs a `thread_ts`

— a value you can only get from a *previous* call to `get_channel_history`

. context7's `get-library-docs`

needs a library ID that comes from `resolve-library-id`

. These are **pipelined tools**: their required arguments are produced by other tools.

My task synthesizer didn't know that. It generated tasks like "reply to the thread about the outage" — without a thread timestamp. The model, quite sensibly, picked `get_channel_history`

first (to find the thread), or declined. My grader marked both choices wrong.

The model wasn't confused. The model was *right*. The benchmark was grading correct multi-step reasoning as failure — and memory's 93% was the tell all along: its tools are single-step, so it scored fine.

The fix was one constraint in the synthesis prompt: **every task must embed concrete values for every required parameter.** "Reply to thread 1721581200.123456 in #incidents" — now single-shot selection is a fair question. Round 2: context7 38% → 100%, slack 54% → 100%.

If your agent benchmark shows a capable model failing on tools that real users navigate fine, check whether you're asking one-step questions about multi-step tools. In my experience most home-grown "tool selection accuracy" numbers have this bug quietly inflating their failure rates.

## Round 3: does it discriminate?

A benchmark that gives everyone 100% is decoration. So round 3 pointed the fixed eval at firecrawl — 26 tools, the lowest static score in my [36-server scan](/posts/mcp-servers-failing-agents.html). If the eval is measuring something real, a messy catalog should score worse. It did, in two specific ways:

**1. Selection misses landed exactly on the naming collisions static rules had flagged.** 84% selection accuracy, and the 16% wasn't random: `extract`

↔`scrape`

, `agent_status`

↔`check_crawl_status`

, `feedback`

↔`search_feedback`

— the same confusable pairs the static rules (N002, C001) had already flagged from names and descriptions alone. That's the result I most wanted: **static lint predicts live model confusion.** The cheap, free, ten-second scan finds the same failure points as the LLM eval.

**2. Refusal collapsed.** Given deliberately out-of-scope tasks, models refused correctly 100% of the time on small, well-documented catalogs — and **50%** of the time on firecrawl's 26 fuzzy tools. Half the time, the model "found" a plausible-sounding tool and called it anyway. Big vague catalogs don't just cause wrong picks; they cause *action when inaction is correct*, which in production is the scariest failure mode there is. Nobody reviews the agent that confidently did something.

## What three rounds bought me

The whole calibration — three rounds, four servers — cost about $0.60 on a small model. For that I got answers to the three questions any eval must survive:

**Is it fair?** After the synthesis fix, well-designed servers score 100%. Failures now mean something.**Does it discriminate?** Messy catalogs score measurably worse, in interpretable ways.**Is it affordable?**~$0.04–0.2 per server. Running it on every PR is a rounding error.

Most benchmark builders skip straight to leaderboards. The calibration step — deliberately trying to prove your own metric is lying — is cheap, unglamorous, and the only thing separating a measurement from a random number generator with axes.

## The calibration isn't finished — a reader proved it

Within a day of the launch post, a reader ([Mads Hansen, in the dev.to comments](https://dev.to/mads_hansen_27b33ebfee4c9/comment/3bjlc)) pointed out two flaws I hadn't caught, and he's right on both.

First: my outcome taxonomy is still too coarse. "Refusal" currently lumps together a model that *asks a clarifying question* and a model that *declines outright* — and neither is separated from the truly dangerous outcome, confidently calling a plausible-but-wrong tool. Four buckets (correct call / correct refusal / correct clarification / unsafe plausible action) with different weights is strictly better, because their production costs are wildly different.

Second, and subtler: **my synthetic tasks can flatter the schemas that generated them.** The synthesizer reads the catalog to write tasks — so a badly-written catalog produces tasks phrased in its own bad vocabulary. The fix is held-out authoring: derive intents from real integration failures, paraphrase them through a step that never sees tool names, and freeze the test split before touching any descriptions.

Both are now [tracked issues on the repo](https://github.com/TengByte/mcpgrade/issues). Which is the point of publishing your methodology instead of just your leaderboard: readers debug your benchmark the way they'd debug your code.

Full raw numbers and methodology live in the repo: [docs/eval-calibration.md](https://github.com/TengByte/mcpgrade/blob/main/docs/eval-calibration.md). If you build agent benchmarks and have found other systematic unfairness patterns, I want to hear about them — open an issue.

*I build production AI agent integrations at a large tech company; mcpgrade is a personal project. The eval runs on any OpenAI-compatible endpoint — bring your own key.*