{"slug": "evals-for-mcp-servers-slow-is-smooth-smooth-is-fast", "title": "Evals for MCP Servers: Slow is Smooth, Smooth is Fast", "summary": "In a technical blog post, the team behind xplainable, an ML platform, details how they built evaluation harnesses for their MCP server to test multi-step agentic workflows, arguing that early evals accelerate development. The post outlines a nine-step churn model build process and highlights failure modes like silent error compounding and train/serve parity issues, sharing their approach in the xplainable-mcp-server GitHub repository.", "body_md": "Evals often feel like a slowdown, an afterthought in the world of agentic AI development. You’ve shipped the tools, the demo works, the agent does the thing. Why formalise it?\n\nThere’s a saying engrained in my mind from my rowing days: **”slow is smooth, smooth is fast.”** It’s always easier to hold good form than to recover it once you’ve lost it. Evals are the same. The teams that build them early move faster later, because every model release, every prompt tweak, every tool change becomes a measurable event instead of a vibe check.\n\nThis post walks through how we built evals for our MCP server. Not as a prescription, more as a worked example you can steal from: the harness ships inside our [MCP server repo on GitHub](https://github.com/xplainable/xplainable-mcp-server).\n\nWe build [xplainable](https://www.xplainable.io), an ML platform covering data preparation, training, deployment, prediction and optimisation. Our MCP server exposes that lifecycle to agents.\n\nWith that in mind, it’s worth delineating two very different kinds of MCP servers.\n\n**Simple tool calls.** “Send an email to Joe Bloggs about the new functionality.” “Send a batch email to the user cohort captured in the CRM.” One or two tool calls, a clear success condition, largely deterministic. If the tool works once, it mostly works.\n\n**Multi-step workflows.** The agent has to orchestrate a ** process**: many tools, in a meaningful order, where the output of one step feeds the next, and where a plausible-looking transcript can still be wrong.\n\nWe’re firmly in the second category. Our MCP server exposes the full machine learning development lifecycle, and then takes it a step further, providing the tools to create mathematically-driven recommendations from the trained models, abstracting away the need to set up experiments manually.\n\nA single “build me a churn model” request steps through:\n\n3. **Data preparation**\n\n4. **Feature engineering**\n\n5. **Persist** the preparation pipeline (so serving matches training)\n\n6. **Train** the model\n\n7. **Deploy** it\n\n8. **Predict** through the deployment\n\n9. **Create a report** (returns a URL)\n\n10. **Optimise**: generate prescriptions/recommendations (optional)\n\nAs you can see, this is substantially more complex than “send an email.”\n\nIt’s worth pausing on **why** this class of workflow resists the usual testing instincts.\n\n**Every step depends on the last, and errors compound silently.** This is true even *within* a single step: a preprocessing pipeline is itself sequential, where each transformation operates on the output of everything before it. Drop a column at step 2 and the encoding at step 5 sees a different frame; reorder two steps and the same spec produces different data. And the same dependency chain runs through the whole workflow: the preprocessing you apply determines the features the model sees; the features determine what the model learns; what the model learns determines whether the optimiser’s recommendations mean anything. A mistake at step 3 doesn’t fail at step 3. It surfaces, if it surfaces at all, as a subtly worse model at step 6, or nonsense prescriptions at step 10. By then the transcript is forty tool calls long and every one of them returned *200 OK*.\n\n**Train/serve parity is the classic trap.** The transformations applied at training time must be applied *identically* at inference time. If the agent trains on engineered features but the deployment serves raw data (or vice versa), nothing crashes; predictions are just quietly wrong. This bug has haunted human ML teams for a decade; an agent orchestrating it through tools inherits it wholesale, with less visibility.\n\n**The state lives server-side, not in the conversation.** Datasets, preprocessor versions, model versions, deployments: these are artifacts on a platform, referenced by IDs the agent must create, track, and wire together correctly across a long context. Training and report generation run as async jobs behind polling. The model only ever sees narrow slices of this state through tool responses; it can’t inspect the pipeline the way an engineer would.\n\n**“Success” is not a status code.** A model can train without error and be useless. Probabilities can saturate to 0.99 across the board. An optimiser can return “recommendations” that recommend changing nothing, or that change columns the business declared immutable. A prescription to reduce churn by making the customer younger is mathematically valid and operationally absurd. Every one of these looks like a pass if all you check is that the calls succeeded.\n\n**And the whole chain runs end-to-end**, from raw dataset to mathematical recommendations fed back into the business. The surface area for plausible-but-wrong is enormous, and the failure modes aren’t crashes; they’re **silent process failures**: training on raw data instead of the engineered features, deploying a model that saturates every probability, generating recommendations that say “do nothing.”\n\nBefore going further, a caveat I think matters. From the Pydantic Evals docs:\n\nEvals are an Emerging Practice\n\nUnlike unit tests, evals are an emerging art/science. Anyone who claims to know exactly how your evals should be defined can safely be ignored.\n\nI’d echo that. If anyone claims “**this** is how evals must be done,” you can quickly discount the rest of the advice. What follows is what worked for us, for this kind of server. Your failure modes will differ.\n\nIf you’re an AI engineer, you’re probably familiar with the current loop:\n\nNotice how many layers a single change travels through, across three repositories, before an agent ever calls it: API, client wrapper, package release, pin bump, server CI, deployment, consumption. The MCP server generates its tool surface at runtime from the client’s registry, so the new tool appears automatically once the pin lands; the only guard rail at that layer is a test pinning the exact tool count. The unit tests at each layer can all be green while the **end-to-end behaviour** (an agent actually completing a job with the tool) has regressed. That gap between “every layer passes” and “the workflow works” is exactly where evals live.\n\nThen you verify by hand:\n\n4. Check the tool shows up in your consumption method (CLI, Claude Desktop)\n\n5. Do one pass through the dataset/prompt you use for all your testing\n\nThis works, right up until it doesn’t. Two situations break it:\n\nOne pass through one prompt tells you nothing about either.\n\nWe built ours on **pydantic-ai** (to drive the agent) and **pydantic-evals** (to score it). Pydantic Evals follows a code-first approach: datasets, cases, and evaluators are all defined in Python.\n\nNote: the original suggestion was mcp-use (an open-source way to connect any LLM to any MCP server and build custom MCP agents, worth a look). We went with pydantic-ai to keep everything in one ecosystem: its agent, MCP client, and eval framework share types.\n\nThe core loop is small. pydantic-ai speaks MCP natively, so the agent gets your server’s **actual** tool surface. No mocks, no re-declared schemas:\n\n``` python\nfrom pydantic_ai import Agentagent = Agent(    model,                      # \"anthropic:claude-sonnet-4-6\", or anything else    toolsets=[mcp_toolset],     # your MCP server, in-process or over HTTP)result = await agent.run(scenario.task_prompt)\n```\n\nTwo targets, two auth modes.The harness is built to point at the serverin-process(authenticated with an API key from the environment, ideal for fast local iteration and CI) or at thehosted production serverover HTTP (behind the same OAuth flow real users go through). Same scenarios, same evaluators; only the toolset construction changes. Being able to eval the exact deployment users hit, auth and all, is worth wiring up early.\n\nEach scenario is one realistic job with an expected shape:\n\n```\nScenario(    name=\"telco_churn_minimal\",    task_prompt=\"Build and deploy a churn model on the uploaded dataset...\",    expected_stages=[DATA_PREP, PERSIST_PREP, TRAIN, DEPLOY, PREDICT],)\n```\n\nAnd pydantic-evals runs it as a dataset of cases, each scored by evaluators:\n\n``` python\nfrom pydantic_evals import Case, Datasetdataset = Dataset(    cases=[Case(name=f\"{scenario.name}[{i}]\", inputs=scenario) for i in range(k)],    evaluators=[StageEvaluator(scenario.expected_stages), SemanticDetectors(), Efficiency()],)report = await dataset.evaluate(run_scenario)\n```\n\nThe single most useful design decision: evaluators inspect **what actually happened on the platform**, not which tools the agent called. “Did it call train_model?\" is a weak assertion: the call can succeed while the process fails.\n\nTo know what actually happened, each case is bracketed by a session ledger: snapshot the platform before the agent runs, diff after, and tear down whatever the run created. The diff is ground truth for “what did the agent build”, and the teardown keeps the eval team clean between runs (this is a real platform with real state and real quotas, not a sandbox):\n\n```\nclass EvalSession:    \"\"\"Before/after ledger of platform artifacts for one eval case.\"\"\"    def snapshot(self):        self._before = self._list_platform_ids()  # datasets, models, deployments...        def diff(self) -> CreatedArtifacts:        now = self._list_platform_ids()        return CreatedArtifacts(**{k: now[k] - self._before[k] for k in now})        def teardown(self, created: CreatedArtifacts) -> list[str]:        \"\"\"Delete everything the run created; return whatever would not delete.\"\"\"\n```\n\nWith that in place, evaluators can assert against platform state instead of transcript vibes:\n\n``` php\ndef check_train(outcome) -> bool:    \"\"\"Model exists AND was trained on the transformed data:    the train call must reference a preprocessor the agent created.\"\"\"    if not outcome.created.models:        return False    created = set(outcome.created.preprocessors)        return any(            call.name in TRAIN_TOOLS and not call.error            and any(arg in created for arg in leaf_values(call.args))            for call in outcome.tool_calls        )\n```\n\nThat check exists because of a real regression: the agent was happily “training” models on raw data, skipping every engineered feature, and every transcript looked fine. An eval that only counted tool calls would have passed.\n\nAlongside stage checks, we run **semantic detectors** for known failure modes: degenerate prescriptions (recommendations that change nothing), saturated probabilities (every prediction ≈ 1.0), drift in columns that were declared immutable. Each detector is a boolean that fires when the failure is present, and each one encodes a specific silent failure we actually observed. This one exists because a live optimiser run once returned twenty “prescriptions” that all prescribed identical lever values, and everything upstream reported success:\n\n``` php\ndef degenerate_prescriptions(prescriptions: list[dict]) -> bool:    \"\"\"All rows prescribe identical lever values. True = failure.\"\"\"    if len(prescriptions) < 2:        return False    mappings = [prescribed_changes(row) for row in prescriptions]    return bool(mappings[0]) and all(m == mappings[0] for m in mappings[1:])\n```\n\nThese came directly from post-mortems; your list will come from yours.\n\nOne thing that turned out to matter as much as the evals themselves: **structured errors**. We propagate typed error envelopes (Pydantic models) all the way through to the tool response:\n\n```\n[VALIDATION_ERROR] Pipeline compilation failed: 'BinOp' object hasno attribute 'type' - Suggestion: check the transformer spec syntax\n```\n\nReadable errors are important for humans; they’re **just as important** for agents. In our runs, models recover from structured errors and retry correctly, where an opaque 500 used to send them into a doom loop. The eval harness measures this too: failed calls that get retried show up as wasted_calls, so error-message quality becomes a number you can watch.\n\nEach run writes a results JSON: per-case stage assertions, detector flags, step_count, wasted_calls, duration, token usage and dollar cost (OpenRouter reports the cost of every request when you ask for it, so a run's price tag is captured automatically), and the tool-call transcript. A small reporting CLI turns any set of result files into a comparison:\n\n(Dashes in the cost column are runs recorded before cost capture landed; old result files stay comparable.)\n\nThe headline metric is **pass@k**, borrowed from code-generation benchmarks. The idea: agent runs are stochastic. The same model, same prompt, and same tools will sometimes succeed and sometimes fail. So instead of asking “did it work once?”, pass@k asks “if I gave the agent k attempts, what is the probability that at least one of them completes the whole workflow?” pass@1 is the single-shot success rate; pass@5 tells you whether retrying rescues the failures or whether the model just cannot do the task. The gap between the two is informative on its own: a model with pass@1 of 0.6 but pass@5 of 1.0 is flaky but capable, while a model stuck at 0.6 for both has a systematic blind spot that no amount of retrying will fix.\n\nTo compute it we run each scenario n times, count the c successes, and use the standard unbiased estimator rather than naively resampling:\n\n``` php\ndef pass_at_k(n: int, c: int, k: int) -> float:    \"\"\"n samples, c passes: 1 - C(n-c, k) / C(n, k)\"\"\"    return 1.0 - comb(n - c, k) / comb(n, k)\n```\n\n…plus the familiar pass@k curves, per-stage pass-rate profiles (so you can see **where** in the pipeline a model falls over, not just that it did), and step-count distributions.\n\nThe per-stage profile earns its keep quickly. Look at that glm-fullrow again: pass@k of 0.00 sounds like total failure, but the stage columns tell a completely different story. Take a real run of GLM 5.3 over the full ten-stage scenario, four scored attempts, at a captured cost of $8.26. The model went four for four on nine of the ten stages, every single attempt: exploration, data prep, feature engineering, training, deployment, prediction, even optimisation. It failed exactly one stage, every time: surfacing the URL of an asynchronously generated report. The headline metric says “cannot do the workflow”; the profile says “does the entire workflow, except one async-timing interaction it never gets right”. Those demand very different responses. The first sends you model shopping; the second sends you to fix one tool’s job-status ergonomics. This is why a single scalar is not enough for multi-step servers: the shape of the failure is the actionable part.\n\nThe tool-call log makes the same point even more bluntly. Across those four attempts the model made 240 tool calls, and 103 of them, 43%, were to a single endpoint: reports_get_job_status, polled over and over waiting for the async report, with 57 of those calls returning errors. Every other tool on the surface behaved. One endpoint’s ergonomics consumed nearly half the model’s effort:\n\nThe call-by-call timeline shows how each attempt actually died. Every one starts the same way: a long, clean run of workflow calls that carries the model through data prep, training, deployment, prediction and optimisation. Then the report job kicks off and the back half of every attempt dissolves into polling and errors:\n\nThis is where the harness pays for itself. Research keeps showing that a [well-built harness can let a smaller model compete with the frontier](https://arxiv.org/abs/2607.08938), and with output-token prices [spread 25–40x between tiers](https://vinvashishta.substack.com/p/minimum-viable-model-structured-model), the cheap model can burn far more tokens and still come out ahead, right up until the next release makes the comparison obsolete. The only sane response is to make model swaps **cheap to measure**.\n\nBecause pydantic-ai handles provider routing, testing a new model is a CLI flag. OpenRouter support required **zero harness changes**:\n\n```\npython -m evals.run \\  --model anthropic:claude-sonnet-4-6 \\  --model openrouter:z-ai/glm-5.3 \\  --scenario telco_churn_minimal -k 3\n```\n\nA real example from this week, same scenario, same tools, same prompt:\n\nBoth models completed the entire ML lifecycle. One took ~60% more steps and ~75% more wall-clock. That’s the shape of answer you want on release day:\n\nThe intent is to wire this into CI: on a new model release (or a new server version), GitHub Actions runs the eval matrix, logs results, and we check for regressions before anything ships, the same way you’d treat a dependency bump.\n\nThe part nobody writes about. Two things kept our harness from rotting on day one:\n\n**Derive what you can from the server itself.** Our tool classifications (read vs write, train tools, predict tools) come programmatically from the same registry the MCP server uses to build its surface, not hand-maintained lists. When a new endpoint is wrapped and appears on the server, the harness picks it up automatically. The trade-off is real, though: deterministic derivation gives you one surface to maintain, but limited flexibility around edge cases; prompts like to be **seen**, and a scenario will never exercise a tool its prompt doesn’t ask for. We keep one deliberate tripwire: a test that pins the exact tool count, so the surface can never change silently.\n\n**Treat scenarios and prompts as a growing library.** New use cases become new scenarios; new prompt variants become a --prompt axis in the same cross-product as --model. The harness doesn't care which one you're varying.\n\n**And one operational lesson: **eval runs create real artifacts (datasets, models, deployments, reports). Snapshot before, diff after, tear down everything the run created, and run them against a dedicated team/workspace, never one you work in.\n\nThe obvious next step is to point this at the rest of the development practice: the frontend, external services, API design, DB design. If an agent is going to drive development, the same question applies everywhere: **how do you know the multi-step process actually worked?**\n\nThe less obvious step is the feedback loop in reverse: our evals have already reshaped the API. Watching agents fail told us which errors needed structure, which tools needed better affordances, which workflows needed to move server-side. It happened again while writing this post: the harness’s teardown couldn’t delete the reports that eval runs generate, because nothing had ever needed to delete one programmatically. The delete endpoint existed in the API; no client wrapper exposed it. Closing that gap for the evals shipped a new wrapper and a new tool on the agent surface the same afternoon. There’s a whole development lifecycle emerging around refactoring APIs **for** agents, and evals are the instrument that tells you whether the refactor helped.\n\nThe node that deserves the highlight is the data model. Every API affordance an agent needs, a pollable job status, a listable artifact lineage, a clear owner for every object, is only expressible if the database models it. When our agents flailed polling reports_get_job_status, the fix wasn't a prompt tweak; it was making job state a first-class, queryable thing. Agent-facing API design is DB design wearing a costume. The evals are what tell you which part of the schema the agents are tripping over.\n\n**Slow is smooth. Smooth is fast.**\n\n**We build ****xplainable****: explainable ML through a platform, an API, and an MCP server. The server, including the eval harness from this post, is open on ****GitHub****.**\n\n[Evals for MCP Servers: Slow is Smooth, Smooth is Fast](https://pub.towardsai.net/evals-for-mcp-servers-slow-is-smooth-smooth-is-fast-11c726250a5d) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/evals-for-mcp-servers-slow-is-smooth-smooth-is-fast", "canonical_source": "https://pub.towardsai.net/evals-for-mcp-servers-slow-is-smooth-smooth-is-fast-11c726250a5d?source=rss----98111c9905da---4", "published_at": "2026-09-04 04:19:42+00:00", "updated_at": "2026-09-04 04:52:08.871174+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "developer-tools"], "entities": ["xplainable", "xplainable-mcp-server", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/evals-for-mcp-servers-slow-is-smooth-smooth-is-fast", "markdown": "https://wpnews.pro/news/evals-for-mcp-servers-slow-is-smooth-smooth-is-fast.md", "text": "https://wpnews.pro/news/evals-for-mcp-servers-slow-is-smooth-smooth-is-fast.txt", "jsonld": "https://wpnews.pro/news/evals-for-mcp-servers-slow-is-smooth-smooth-is-fast.jsonld"}}