{"slug": "my-eval-said-a-perfect-mcp-server-was-broken-it-was-the-eval-that-was-lying", "title": "My eval said a perfect MCP server was broken. It was the eval that was lying.", "summary": "A developer building mcpgrade, an MCP server evaluation tool, discovered that its LLM-powered eval was producing false failure rates because it generated single-step tasks for tools requiring multi-step pipelines. After fixing the task synthesis to embed concrete parameter values, the tool selection accuracy for the context7 server jumped from 38% to 100% and for server-slack from 54% to 100%. The corrected eval also confirmed that static linting predicts live model confusion, with firecrawl's 26-tool catalog scoring 84% selection accuracy and 50% refusal accuracy.", "body_md": "# My eval said a perfect MCP server was broken. It was the eval that was lying.\n\nWhen I added an LLM-powered eval to [mcpgrade](https://github.com/TengByte/mcpgrade), the first real run produced a result that looked like a scoop: context7 — a server with a *perfect* static score — failed tool selection 62% of the time. A model shown its two-tool catalog picked the \"wrong\" tool on 5 of 8 tasks.\n\nIf I had shipped that number, it would have been wrong. Not slightly wrong — systematically, unfairly wrong. This post is about how I caught it, because the failure mode generalizes to most agent benchmarks people are building right now.\n\n## The setup\n\nmcpgrade's `--eval`\n\nmode works like this: it reads a server's tool catalog, synthesizes realistic single-step tasks (\"find the Slack channel where the incident was discussed\"), shows a model the full catalog, and measures three things — does it pick the right tool, does it fill valid arguments, and does it correctly *refuse* tasks that no tool can handle.\n\nRound 1, on three real servers, cost about twelve cents and produced this:\n\n| Server | Static score | Tool selection | Args | Refusal |\n|---|---|---|---|---|\n| context7 (2 tools) | 100 | 38% |\n100% | 100% |\n| server-memory (9 tools) | 81 | 93% | 100% | 100% |\n| server-slack (8 tools) | 97 | 54% |\n100% | 100% |\n\nTwo servers with excellent static scores, apparently failing live. Either static analysis was worthless, or the eval was broken.\n\n## The eval was broken\n\nEvery \"miss\" traced to one cause. Slack's `post_message`\n\nneeds a `thread_ts`\n\n— a value you can only get from a *previous* call to `get_channel_history`\n\n. context7's `get-library-docs`\n\nneeds a library ID that comes from `resolve-library-id`\n\n. These are **pipelined tools**: their required arguments are produced by other tools.\n\nMy task synthesizer didn't know that. It generated tasks like \"reply to the thread about the outage\" — without a thread timestamp. The model, quite sensibly, picked `get_channel_history`\n\nfirst (to find the thread), or declined. My grader marked both choices wrong.\n\nThe model wasn't confused. The model was *right*. The benchmark was grading correct multi-step reasoning as failure — and memory's 93% was the tell all along: its tools are single-step, so it scored fine.\n\nThe fix was one constraint in the synthesis prompt: **every task must embed concrete values for every required parameter.** \"Reply to thread 1721581200.123456 in #incidents\" — now single-shot selection is a fair question. Round 2: context7 38% → 100%, slack 54% → 100%.\n\nIf your agent benchmark shows a capable model failing on tools that real users navigate fine, check whether you're asking one-step questions about multi-step tools. In my experience most home-grown \"tool selection accuracy\" numbers have this bug quietly inflating their failure rates.\n\n## Round 3: does it discriminate?\n\nA benchmark that gives everyone 100% is decoration. So round 3 pointed the fixed eval at firecrawl — 26 tools, the lowest static score in my [36-server scan](/posts/mcp-servers-failing-agents.html). If the eval is measuring something real, a messy catalog should score worse. It did, in two specific ways:\n\n**1. Selection misses landed exactly on the naming collisions static rules had flagged.** 84% selection accuracy, and the 16% wasn't random: `extract`\n\n↔`scrape`\n\n, `agent_status`\n\n↔`check_crawl_status`\n\n, `feedback`\n\n↔`search_feedback`\n\n— the same confusable pairs the static rules (N002, C001) had already flagged from names and descriptions alone. That's the result I most wanted: **static lint predicts live model confusion.** The cheap, free, ten-second scan finds the same failure points as the LLM eval.\n\n**2. Refusal collapsed.** Given deliberately out-of-scope tasks, models refused correctly 100% of the time on small, well-documented catalogs — and **50%** of the time on firecrawl's 26 fuzzy tools. Half the time, the model \"found\" a plausible-sounding tool and called it anyway. Big vague catalogs don't just cause wrong picks; they cause *action when inaction is correct*, which in production is the scariest failure mode there is. Nobody reviews the agent that confidently did something.\n\n## What three rounds bought me\n\nThe whole calibration — three rounds, four servers — cost about $0.60 on a small model. For that I got answers to the three questions any eval must survive:\n\n**Is it fair?** After the synthesis fix, well-designed servers score 100%. Failures now mean something.**Does it discriminate?** Messy catalogs score measurably worse, in interpretable ways.**Is it affordable?**~$0.04–0.2 per server. Running it on every PR is a rounding error.\n\nMost benchmark builders skip straight to leaderboards. The calibration step — deliberately trying to prove your own metric is lying — is cheap, unglamorous, and the only thing separating a measurement from a random number generator with axes.\n\n## The calibration isn't finished — a reader proved it\n\nWithin a day of the launch post, a reader ([Mads Hansen, in the dev.to comments](https://dev.to/mads_hansen_27b33ebfee4c9/comment/3bjlc)) pointed out two flaws I hadn't caught, and he's right on both.\n\nFirst: my outcome taxonomy is still too coarse. \"Refusal\" currently lumps together a model that *asks a clarifying question* and a model that *declines outright* — and neither is separated from the truly dangerous outcome, confidently calling a plausible-but-wrong tool. Four buckets (correct call / correct refusal / correct clarification / unsafe plausible action) with different weights is strictly better, because their production costs are wildly different.\n\nSecond, and subtler: **my synthetic tasks can flatter the schemas that generated them.** The synthesizer reads the catalog to write tasks — so a badly-written catalog produces tasks phrased in its own bad vocabulary. The fix is held-out authoring: derive intents from real integration failures, paraphrase them through a step that never sees tool names, and freeze the test split before touching any descriptions.\n\nBoth are now [tracked issues on the repo](https://github.com/TengByte/mcpgrade/issues). Which is the point of publishing your methodology instead of just your leaderboard: readers debug your benchmark the way they'd debug your code.\n\nFull raw numbers and methodology live in the repo: [docs/eval-calibration.md](https://github.com/TengByte/mcpgrade/blob/main/docs/eval-calibration.md). If you build agent benchmarks and have found other systematic unfairness patterns, I want to hear about them — open an issue.\n\n*I build production AI agent integrations at a large tech company; mcpgrade is a personal project. The eval runs on any OpenAI-compatible endpoint — bring your own key.*", "url": "https://wpnews.pro/news/my-eval-said-a-perfect-mcp-server-was-broken-it-was-the-eval-that-was-lying", "canonical_source": "https://tengli.dev/posts/calibrating-mcp-evals.html", "published_at": "2026-07-27 00:00:00+00:00", "updated_at": "2026-07-28 01:25:24.410219+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "developer-tools"], "entities": ["mcpgrade", "context7", "server-memory", "server-slack", "firecrawl"], "alternates": {"html": "https://wpnews.pro/news/my-eval-said-a-perfect-mcp-server-was-broken-it-was-the-eval-that-was-lying", "markdown": "https://wpnews.pro/news/my-eval-said-a-perfect-mcp-server-was-broken-it-was-the-eval-that-was-lying.md", "text": "https://wpnews.pro/news/my-eval-said-a-perfect-mcp-server-was-broken-it-was-the-eval-that-was-lying.txt", "jsonld": "https://wpnews.pro/news/my-eval-said-a-perfect-mcp-server-was-broken-it-was-the-eval-that-was-lying.jsonld"}}