{"slug": "a-go-no-go-rubric-for-evaluating-mcp-servers-before-they-touch-production", "title": "A Go/No-Go Rubric for Evaluating MCP Servers Before They Touch Production Traffic", "summary": "A developer published a two-tier go/no-go rubric for deciding whether an MCP server is ready to be wired into production agents, arguing that protocol conformance tests and agent-level evals measure different failure surfaces. Tier 1 covers six deterministic protocol checks — schema validity, structured errors, negative tests, distinguishable timeouts and partial failures, per-call token validation, and measured p95 latency under load — and the author advises stopping if a server scores below 5/6 before running Tier 2 agent evals. Tier 2 requires the eval harness to use the same transport, auth path, and tool list as production, and to score tool selection separately from argument construction.", "body_md": "Most MCP \"readiness checklists\" are really just a list of things that would be nice to have. They tell you to add logging, add auth, add tests — true, but not decision-grade. None of them answer the question a lead engineer actually asks before a release: *given everything we know right now, do we ship this server to production agents or not?*\n\nThis post is a rubric, not a wishlist. It's built to produce a number and a decision, the way a real go/no-go review works. It won't re-teach OAuth flows or walk through building a server from scratch — we've covered both of those in depth already. This is specifically about **evaluation**: how you score a server you (or someone else) already built, before it gets wired into an agent that real users depend on.\n\nProtocol-level tests (does `tools/list` return valid schemas, does the server respond to `initialize`) tell you the server speaks MCP correctly. They don't tell you what happens when an agent is calling it. Those are two different failure surfaces, and conflating them is the single most common mistake teams make when they say a server is \"tested.\"\n\nServer tests validate implementation behavior against known requirements, and MCP evals measure agent performance across realistic tasks. A server can pass every unit and conformance test and still fall apart the moment a non-deterministic agent has to choose between five similar-sounding tools, or retries a call that already partially succeeded. An MCP server can pass every protocol and handler test and still break down once an agent has to pick a tool, fill in valid arguments, and work through errors on the way to finishing a task. A scorer that reads only the final response misses silent failures and duplicate records left behind by a retry.\n\nSo the rubric below has two tiers, and you need both before you can call a server production-ready:\n\nRun these against the raw protocol, no LLM involved. They should be fast, deterministic, and part of CI — not something you eyeball once before a release.\n\n| # | Check | Why it matters | \n|---|---|---|\n| 1 | Every tool's JSON schema is valid and matches the runtime handler's actual accepted inputs | Schema drift is invisible until an agent sends a payload the handler silently mishandles | \n| 2 | Malformed input returns a structured error, not a stack trace or empty 200 | Agents parse error text to decide their next move; garbage in, garbage reasoning out | \n| 3 | Every documented tool has at least one negative test (bad auth, missing field, out-of-range value) | Happy-path-only test suites are the norm, and it shows in production incident logs | \n| 4 | Timeouts and partial failures are distinguishable from success | An agent that can't tell \"still running\" from \"failed\" will duplicate side effects | \n| 5 | Token/session validation happens on every call, not just at connection time | Long-lived sessions with front-loaded auth checks are a common gap once bearer tokens get forwarded | \n| 6 | p95 latency under realistic load is measured, not assumed | A tool that's 200ms in dev and 5s under load stalls the agent's reasoning loop and can trigger retries that compound the problem | \n\nIf you're scoring under 5/6 here, stop — don't move to Tier 2 yet. Agent-level evals on top of a shaky protocol layer just produce noisy, non-repeatable results.\n\nThis is the part almost nobody automates, because it requires actually running an agent against the server rather than curling endpoints.\n\n| # | Check | Why it matters | \n|---|---|---|\n| 7 | Harness uses the *same* transport, auth path, and tool list as production | The eval harness acts as an MCP client, so it should use the same transport, authentication path, and exposed tools as the client used in production. A harness that skips OAuth or substitutes a shortened tool list is measuring a different configuration, and its scores say nothing about how reliably agents will use the production server. | \n| 8 | Tool selection is scored separately from argument construction and outcome | A single pass-or-fail score shows a run failed but does not identify the cause. Effective agent evaluation separates tool use, arguments, execution, and outcome into distinct measurements. | \n| 9 | Every task is run multiple trials, not once | Repeated trials: Multiple runs of the same task that reveal how consistently the agent completes it. A repeatable MCP eval ties each task-level pass rate to the agent behavior and system state behind it. | \n| 10 | Adversarial and near-miss prompts are included, not just the intended use case | Test with adversarial prompts first. Don't just test the happy path. Try prompts that are similar to the intended use case but shouldn't activate the server. | \n| 11 | System state after a run is checked, not just the final agent message | A trajectory can \"look\" successful while leaving duplicate writes or half-completed operations behind | \n| 12 | Recovery behavior is tested, not just failure detection | Being observable isn't the same as being recoverable — check whether state can actually roll back or resume after a partial failure | \n\nScore Tier 2 the same way: 0/1 per row, and treat anything under 5/6 as \"not yet,\" not \"close enough.\"\n\nA server that scores 4/6 and 3/6 isn't broken — it's early. The honest move is to say so publicly (or at least in your internal docs): note what's covered, what isn't, and what an integrator should assume is their responsibility until it's fixed. If the server has no concept of budgets, quotas, or side-effect governors, then it is asking the orchestrator to do all the defensive work. Sometimes that's acceptable. But then be honest: the server is not production-complete on its own. That kind of scoped honesty is more useful to the next engineer than a README that just says \"production ready\" with no evidence behind it.\n\nThe table above is something you can run once for a demo. The harder part — and the part that actually matters eight weeks from now when the server has changed twelve times — is making it repeatable: fixed task sets, versioned scorers, a way to diff \"did this get better or worse after the last deploy\" instead of re-litigating the whole checklist from memory every release.\n\nThat's exactly the gap our **MCP Production Checklist + Eval Templates pack** is built to close. It's the scored version of everything above, packaged as reusable YAML task definitions and a minimal harness scaffold you can point at any server — plus the [Production Checklist](https://quietdesk15.gumroad.com/l/mcpevalpack) itself as a standalone document your team can actually use in a release review instead of reconstructing it from a blog post every time.\n\nIf you're evaluating a server you didn't build — a community package, a vendor's offering, or a teammate's first attempt — running both tiers above before it touches real traffic is the difference between finding the gaps in a review meeting and finding them in an incident channel at 2am.\n\n*Written with AI assistance and reviewed for accuracy.*", "url": "https://wpnews.pro/news/a-go-no-go-rubric-for-evaluating-mcp-servers-before-they-touch-production", "canonical_source": "https://dev.to/quietdesk_studio_83466628/a-gono-go-rubric-for-evaluating-mcp-servers-before-they-touch-production-traffic-a0d", "published_at": "2026-09-26 22:44:18+00:00", "updated_at": "2026-09-26 23:30:53.091250+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "mlops", "ai-tools", "developer-tools"], "entities": ["MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-go-no-go-rubric-for-evaluating-mcp-servers-before-they-touch-production", "markdown": "https://wpnews.pro/news/a-go-no-go-rubric-for-evaluating-mcp-servers-before-they-touch-production.md", "text": "https://wpnews.pro/news/a-go-no-go-rubric-for-evaluating-mcp-servers-before-they-touch-production.txt", "jsonld": "https://wpnews.pro/news/a-go-no-go-rubric-for-evaluating-mcp-servers-before-they-touch-production.jsonld"}}