{"slug": "what-is-an-mcp-eval-why-your-server-passes-every-test-and-still-fails", "title": "What Is an MCP Eval? Why Your Server Passes Every Test and Still Fails", "summary": "An engineer who built an evaluation engine for arbitrary MCP servers explains that an MCP eval is a realistic task a model must complete using only the server's tools, distinguishing it from unit tests that only verify protocol conformance. The post highlights that the signature failure is a wrong answer produced entirely from successful calls, which no inspector catches, and outlines four useful outcomes: pass, wrong answer, too many calls, and untestable. It also notes that call count degrades before pass rate does, making it an earlier warning signal.", "body_md": "📖 TL;DR\n\nAn MCP eval is a realistic task a model must complete using only your server's tools— not an assertion about one call.A test asks \"did the call work?\". An eval asks \"could an agent get the right answer?\"The signature failure is an answer that is wrong even though every call returned 200.No inspector catches it.There are four useful outcomes: pass, wrong answer, too many calls, and untestable. The last one is not a failure.Evals are not deterministic, and pretending otherwise is how you get a number you cannot trust.Call count degrades before pass rate does, which makes it the earlier warning signal.\n\nYou built an MCP server. The handshake works, every tool returns valid JSON, your integration tests are green. Then someone connects it to Claude and it is useless.\n\nThat gap has a name now. **An MCP eval is what closes it.**\n\nThe distinction is simple once you see it. Your test suite proves the protocol works. It says nothing about whether a model can use what you built.\n\nThose are genuinely different questions, and only one of them is the reason your server exists.\n\nThe [MCP specification](https://modelcontextprotocol.io/) has nothing to say about the second one, and it should not — conformance is not usability. The [2026-07-28 release](https://blog.modelcontextprotocol.io/posts/2026-07-28/) tightened the protocol considerably and did not change this at all.\n\nI have spent the last few months building an eval engine for arbitrary MCP servers. This post covers what an eval is, the four outcomes worth distinguishing, why the results are non-deterministic, and when you should bother.\n\n**An MCP eval is a task, phrased the way a user would ask it, that a model has to complete using only your server's tools.**\n\nNot an assertion. Not a mocked conversation. A question, and a score for whether the model got there.\n\nA test looks like this:\n\n```\nassert(callTool(\"list_issues\", { status: \"open\" }).length > 0)\n```\n\nAn eval looks like this:\n\n```\nTask:        \"Which open issue in the billing project has been\n              waiting longest, and who is it assigned to?\"\nBudget:      4 tool calls\nExpectation: names a specific issue and an assignee\n```\n\nNothing in the eval names a tool. **Choosing the tool is the thing being tested.**\n\nThe model gets your tool list, your descriptions and your schemas, and has to work out the rest. That is exactly what happens in production.\n\nThese three get used interchangeably and they measure different things.\n\n| Answers | Misses | |\n|---|---|---|\nInspecting |\nDoes the handshake succeed? Do tools list? | Everything about usability |\nTesting |\nDoes this call, with these args, return what I expect? | Whether a model would ever make that call |\nEvaluating |\nCan an agent reach the right answer from my descriptions? | Determinism, exhaustive coverage |\n\nThe key structural point: **when you write a test, you have already made the choice the model has to make.**\n\nYou picked the tool. You picked the arguments. You skipped the only step that can fail in the way that matters.\n\nAll three are worth having. [Testing catches protocol and correctness bugs](https://mcpplaygroundonline.com/blog/how-to-test-mcp-servers-step-by-step), and it is faster and cheaper than an eval. Evals catch a class of defect testing structurally cannot reach.\n\nHere is the scenario that made me build this.\n\nA docs server exposes `search`\n\nand `get_page`\n\n. A user asks which regions a product supports.\n\nThe agent calls `search(\"regions\")`\n\n. It gets back five ranked results. It calls `get_page`\n\non the first one. That page mentions two regions in an example snippet.\n\nThe agent answers: \"It supports us-east and eu-west.\"\n\n**Every call succeeded.** Valid JSON, no errors, no timeouts. Your logs are clean and your dashboard is green.\n\nThe real answer was eleven regions, listed on a page the search ranked fourth.\n\nThis is the defect evals exist to findA wrong answer produced entirely from successful calls. There is no error to catch, no exception to log, no status code to alert on.\n\nThe only way to detect it is to check the answer.\n\nNothing in your stack is lying. The protocol worked perfectly. The server was just hard to use, and the model did what it could.\n\nA binary pass/fail throws away most of the signal. Four outcomes are worth distinguishing.\n\nThe agent answered, stayed within its call budget, and the answer was correct. **This is the only outcome that requires judging content**, which means it is the only one a model decides.\n\nThe agent produced an answer and it was not right. Either it could not find the information, or your tools gave it something misleading.\n\nThis is the outcome from the scenario above, and it is the most valuable one an eval produces.\n\nThe agent got there, but it took eight calls when the budget was four. **This is the tool-description signal**, and it is the one to watch over time.\n\nThe agent was guessing. It tried a tool, got something unhelpful, tried another. Every one of those calls cost you latency, tokens and — if your API is metered — money.\n\nCall count degrades before pass rate does. A task that passed in two calls last month and passes in six today is a regression, even though the number in the pass column did not move.\n\nI covered the root cause of this in [why 97% of MCP tool descriptions are broken](https://mcpplaygroundonline.com/blog/mcp-tool-description-quality).\n\nSomething outside your server broke. The connection dropped, the driver model rate-limited, the harness fell over.\n\n**This is not a failure and it must never be counted as one.** It gets its own bucket for a reason I will come to.\n\n**See a finished eval run** — tools detected, tasks written against them, a model made to complete each one, and a report showing where it went wrong. No sign-up: [Walk through MCP Evals →](https://mcpplaygroundonline.com/mcp-evals)\n\nThis is the design decision I would defend hardest, and it took me a while to get right.\n\n**A failure only means something if you know the input was valid.**\n\nSay an eval calls `get_issue(\"PROJ-4821\")`\n\nand gets a 404. Is that a defect in your server?\n\nIt depends entirely on where that ID came from.\n\n`list_issues`\n\nreturned it a second earlierSame error code, opposite meaning. **The only thing that separates them is provenance.**\n\nThis is why a serious eval engine harvests real values from your server before planning anything that needs them. Failures on harvested values are defects. Failures on invented values are untestable.\n\nCollapsing those two into \"fail\" gives you a report full of noise that developers correctly learn to ignore.\n\nModels write the tasks. A model drives the tools. A model grades the answer. **Run the same suite twice and you can get different numbers.**\n\nPeople find this disqualifying. I think that reaction comes from expecting an eval to be a test, which it is not.\n\nThe research benchmarks live with the same constraint. [MCP-Atlas](https://arxiv.org/abs/2602.00933) grades 1,000 tasks across 36 real servers using a rubric-driven judge, and [MCP-Bench](https://openreview.net/forum?id=fe8mzHwMxN) pairs rule-based checks with LLM scoring for exactly this reason. **Nobody has found a way to make this deterministic**, because the thing being measured is not.\n\nAn eval is *evidence about how an agent behaves against your server*. It is not a pass/fail certificate, and any tool that presents it as one is overselling.\n\nTwo rules make non-determinism workable:\n\n**Reproduce before you act.** A single failure is a lead, not a finding. Run it again. A defect that reproduces is real; one that does not is variance.\n\n**Read the transcript, not the score.** The number tells you where to look. The transcript — which tools were called, with what arguments, what came back — tells you whether the verdict was fair.\n\nThere is a third rule I only arrived at by getting it wrong: **the grader must fail open.**\n\nIf the judge model errors out or returns nothing for an item, that item stays passing. A missing verdict is not evidence of a defect. Failing closed would let an unrelated API hiccup invent bugs in your server.\n\nEvals cost model calls. They are slower and pricier than tests. Some servers do not need them.\n\n**Worth it when:**\n\n**Probably not yet when:**\n\nThat last one matters more than it looks. **Fix your annotations before you run evals**, or safe tooling will skip most of your tools. The [spec repository](https://github.com/modelcontextprotocol/modelcontextprotocol) documents all three hints, and they take minutes to add.\n\nAn eval calls tools with arguments designed to succeed. An unannotated `delete_record`\n\nwould really delete a record, so anything responsible has to treat unannotated tools as destructive.\n\nStart smaller than you think.\n\n**Pick your three most-used tools.** Write three tasks a real user would ask, in their words, with no tool names in them.\n\nConnect a model to your server and give it each task. [Any of the models available in the browser](https://mcpplaygroundonline.com/blog/testing-mcp-servers-with-real-ai-models) will do for a first pass — you do not need a harness to learn something.\n\nWatch three things: did it answer, was the answer right, and how many calls did it take.\n\nYou will almost always find one of two things. A tool the model never reaches for, or a pair it keeps confusing. Both are description problems, and both are fixable in an afternoon.\n\nOnce that loop is useful, make it repeatable so a description change gets checked instead of hoped about.\n\n**What is an MCP eval?**\n\nAn MCP eval is a task a model has to complete using only your MCP server's tools. Rather than checking that a tool returns HTTP 200, it checks whether an agent can work out which tool to call from your names and descriptions, and whether the answer it produces is correct.\n\n**How is this different from testing with an MCP client?**\n\nA client tells you the protocol works — the handshake succeeds, the schema validates, the call returns. An eval tells you the server is usable: that a model given your tools can reach the right answer without burning calls guessing. The most common failure it finds is an answer that is wrong even though every call succeeded.\n\n**Are MCP eval results deterministic?**\n\nNo. Models write, drive and grade the evals, so the same suite can come out differently on a second run. Results are evidence about how an agent behaves against your server, not a fixed pass/fail certificate. Reproduce a failure before acting on it, and read the transcript rather than the score.\n\n**Do MCP evals replace my test suite?**\n\nNo. Tests are faster, cheaper and deterministic, and they catch protocol and correctness bugs an eval would waste money rediscovering. Evals catch a different class of defect that tests structurally cannot reach. Run both.\n\n**What does an untestable eval outcome mean?**\n\nIt means something outside your server broke — the connection dropped, the driver model rate-limited, or the harness failed. It is not a failure and must not be counted as one. Separating untestable from failed is what keeps an eval report free of noise developers learn to ignore.\n\n**Which eval outcome should I watch most closely over time?**\n\nCall count. A task that passed in two calls last month and passes in six today is a regression even though the pass column did not move. Call count degrades before pass rate does, which makes it the earliest signal that tool descriptions have drifted.\n\n**Why do missing tool annotations matter for MCP evals?**\n\nA functional eval calls tools with arguments designed to succeed, so an unannotated destructive tool would really run. Safe tooling has to treat an unannotated tool as destructive and skip it. Declaring `readOnlyHint`\n\n, `idempotentHint`\n\nand `destructiveHint`\n\nbuys back that test coverage.\n\nA test proves your server responds. **An eval proves it can be used.**\n\nThe gap between those is where the expensive failures live — wrong answers assembled from successful calls, and agents burning four extra calls because two of your tools read alike.\n\nStart with three tasks and three tools. Watch the call count as closely as the pass rate. Reproduce before you act.\n\n**Try it on a real server.** Connect any MCP server in the browser and give a real model a task. No install, no config file.\n\n*Originally published on MCP Playground.*", "url": "https://wpnews.pro/news/what-is-an-mcp-eval-why-your-server-passes-every-test-and-still-fails", "canonical_source": "https://dev.to/rupa_tiwari_dd308948d710f/what-is-an-mcp-eval-why-your-server-passes-every-test-and-still-fails-41gf", "published_at": "2026-08-17 12:10:05+00:00", "updated_at": "2026-08-17 12:43:06.863847+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools", "machine-learning"], "entities": ["MCP", "Claude"], "alternates": {"html": "https://wpnews.pro/news/what-is-an-mcp-eval-why-your-server-passes-every-test-and-still-fails", "markdown": "https://wpnews.pro/news/what-is-an-mcp-eval-why-your-server-passes-every-test-and-still-fails.md", "text": "https://wpnews.pro/news/what-is-an-mcp-eval-why-your-server-passes-every-test-and-still-fails.txt", "jsonld": "https://wpnews.pro/news/what-is-an-mcp-eval-why-your-server-passes-every-test-and-still-fails.jsonld"}}