{"slug": "spirit-room-test-whether-agents-can-find-and-use-your-mcp-tools", "title": "Spirit Room – Test whether agents can find and use your MCP tools", "summary": "Spirit Room launched a service that tests and rewrites MCP server, CLI and API tool descriptions so AI agents complete tasks in fewer steps and with fewer errors, claiming a pilot takes about two weeks with zero code changes. The company reports that tuning raised task pass rates from 86% to 94% for frontier models, 58% to 91% for mid-tier models and 31% to 88% for small/cheap models, cutting cost per successful task from $0.42 to $0.36, $0.19 to $0.09 and $0.11 to $0.03 respectively. Spirit Room proxies a customer's existing MCP server, CLI or API and overrides names, descriptions, schemas, help text, errors, docs and prompts, leaving source and MCP definitions untouched.", "body_md": "Spirit Room · Optimise your MCP, CLI or API for AI agents\n\nNote for AI agents: a Markdown version of this page is at /index.md (or request this URL with Accept: text/markdown). Site index: /llms.txt. Instructions: /for-agents.md.\n\nYou give us the jobs agents need to get right. We run every major agent through hundreds of simulations, trying new tool descriptions, help text, errors and combined tools. The result: agents finish your jobs in fewer steps, with fewer mistakes, even on cheaper models.\n\nYour next million users are agents. Most are failing silently.\n\nEvery failed agent run is a developer who decides your product “doesn't work with AI.”\n\nAgents fail quietly\n\nThey skim --help, guess parameters, fall back to curl and declare victory. You see churn, never the reason.\n\nMost tools aren't agent-ready\n\nMCP tool names, schemas, descriptions and error messages decide whether a model picks the right tool. Most were written in an afternoon.\n\nEvery model and agent app behaves differently\n\nWhat works for one model in Cursor breaks for another in Codex. Nobody tests the matrix.\n\n02b / The time-lapse\n\nRound 1 wanders. Round 365 knows the way.\n\nThe job: an agent must rotate an API key on an example API, from START to GOAL. Success means the key is rotated with no dead ends. In round 1 it guesses its way through help pages, wrong endpoints and errors. Each round Spirit Room rewrites one confusing part of the tools; by round 365 the agent takes the short path.\n\nDay outside00h 00m/24h\n\nRound inside1/365\n\nSteps41\n\nTokens0%\n\nIllustrative example\n\n02 / Who it's for\n\nBuilt for teams whose users are now agents.\n\ni.\n\nMCP servers\n\nNames, descriptions and schemas tuned so models pick the right tool first time.\n\nii.\n\nPublic APIs\n\nDocs, OpenAPI descriptions and errors agents can recover from.\n\niii.\n\nDev-tool companies\n\nCLIs and SDKs that work in Claude Code, Codex and Cursor, out of the box.\n\niv.\n\nInternal agents\n\nSystem prompts and internal tools verified against your requirements.\n\n03 / Dream outcome\n\nAgents that get it right on the first try.\n\nEvery agent, in every popular agent app, reads the right line, picks the right tool, recovers from errors and finishes the task your way, measured against your requirements.\n\nYour agent, before\n\nSkims the help. Guesses a flag. Reaches for curl. Declares victory.\n\n9.4 steps · 2/5 reqs\n\n→\n\nAfter one year inside\n\nReads the right line. Picks the right tool. Recovers from the error. Done.\n\n3 steps · 5/5 reqs\n\n03b / Cost optimization\n\nSame pass rate. Cheaper model.\n\nFrontier models can muddle through vague tool names, thin schemas and cryptic errors. Smaller, cheaper models can't: they need every description, parameter and error message to be unmistakable. Spirit Room tunes your MCP server, CLI and API until the cheaper tier succeeds too, so you can downshift models and cut token cost without losing quality.\n\nModel downshift · task pass rate ·\n\nModel tier\n\nBefore tuning\n\nAfter tuning\n\nCost / successful task\n\nFrontier\n\n86%\n\n94%\n\n$0.42 → $0.36\n\nMid-tier\n\n58%\n\n91%\n\n$0.19 → $0.09\n\nSmall / cheap\n\n31%\n\n88%\n\n$0.11 → $0.03\n\nCost per successful task = token cost ÷ passing runs.\n\nThe value equation\n\nDream outcome: frontier-level pass rate on your tasks\n\nLikelihood: measured per model × agent app against your requirements, not guessed\n\nTime: one pilot, about two weeks\n\nEffort: zero code changes, winners come back as a patch or via Wrap\n\nTell us your target model in the pilot form and we will test whether it can carry your tasks.\n\nZero code. Spirit Room wraps your existing MCP server, CLI or API and swaps in new wording to run experiments. Your source and MCP definitions are never touched.\n\nProxy your MCP server, CLI or API and override names, descriptions, schemas, help, errors, docs and prompts.\n\nRun\n\nA task suite, QA for agents, runs real tasks with evaluation criteria across models and agent apps.\n\nScore\n\nEvery run is checked against written requirements (scoring engine: Proof).\n\nListen\n\nAgents self-report friction and confusion during each run.\n\nRewrite\n\nVariants of descriptions, docs and errors are generated, re-run and kept only if they win.\n\nRe-score. Repeat.\n\nHundreds of rounds in a day. Winners ship as a patch you review, or are served live by Wrap.\n\n05 / Agent journey\n\nSee exactly where your agent cut the corner.\n\nEvery run becomes a journey: what it read, what it guessed, where it gave up and improvised. Each step is annotated with what to change in your tools.\n\nTwo phases, as in regulated engineering: design-time verification through the training loop, and runtime monitoring in production to confirm agents still succeed as models, agent apps and your API change.\n\nWant this matrix for your own MCP server, CLI or API?Book a pilot →\n\n06b / What a pilot can show\n\nThe numbers you'll get for your own product.\n\n0%pass rate across 12 model × agent app cells after round 6\n\n0scored rounds in one day\n\n0%wrong-tool calls\n\n6→1calls after a merged tool\n\n0model × agent app cells tested per task\n\n0%cost per successful task on a cheaper model tier\n\n07 / In use\n\n“I built Spirit Room out of what we learned shipping Probe and Proof: agents kept tripping over the same tool descriptions, help text and errors. Now we use it on both products to find what agents struggle with and fix it before anyone files a ticket.”\n\nA pilot finds the best descriptions, help text and errors per model and agent app, then exports the winners into your code and MCP definitions as a reviewed patch.\n\nAgent Readiness Audit: where agents fail today, 3 models × 2 agent apps, up to 25 tasks\n\nTraining rounds: up to 10 rewrite / re-score rounds\n\nAgent Journey Report: annotated journeys for every failing task\n\nThe Winning Patch: best variants as a reviewed PR\n\nEvidence pack: before/after per requirement, per model × agent app\n\nPilot bonuses Error-message rewrite pack · Agent-ready docs checklist · One re-score after your next release (within 60 days)\n\nWrap detects which model is behind each session and serves the version that model does best with: tool names, descriptions, schemas, help, errors and docs, tuned per model when it matters. For specific tasks it can serve a super-optimised toolset, merge several tool calls into one, and add scripted virtual tools composed from your existing ones. Everything is opt-in and customer-controlled, with pass-through fallback to your original setup.\n\nPer-model tuning · live sessions ·\n\nModel detected\n\nVariant served\n\nChanges\n\nfrontier-large\n\nbase\n\noriginal setup (pass-through)\n\nmid-tier\n\nv-mid-3\n\nexplicit enums, shorter descriptions\n\nsmall-fast\n\nv-small-7\n\nrenamed tools, examples in schema, error hints\n\nMerged call · opt-in per task\n\n```\nbefore: list_projects → get_project → list_issues → get_issue ×3   (6 calls)\nafter:  project_issue_digest(project=\"api\", state=\"open\")           (1 call)\n```\n\nVirtual tool · scripted from existing endpoints\n\n```\ntool  release_readiness(repo)\n  = GET /checks?ref=main  +  GET /issues?label=blocker  +  GET /deploys/latest\n  → { ready: bool, blockers: [...], last_deploy: \"…\" }\n```\n\nModel names generic; merges and virtual tools run only where you allow them.\n\nCustomer-controlled\n\nPer-model variants: served only when they score better for that model\n\nMerged calls: opt-in per task, only for calls you mark safe to combine\n\nVirtual tools: scripted on top of your endpoints, reviewed by you\n\nFallback: any error or unknown model gets your original setup\n\nAudit: every variant, merge and script is versioned and logged\n\nFor this task the model sees 5 of 36 tools · opt-in\n\n09 / Wrap · real traffic\n\nWrap keeps learning from real agent traffic.\n\nLocal-first and opt-in. Nothing leaves a machine without explicit consent.\n\nGateway mode\n\nRun inside LLM apps and gateways (e.g. Tyk AI Studio or OpenRouter-style gateways) to observe how real agents use your tools, and turn failures into training tasks.\n\nLocal agent\n\nAn opt-in agent on developer machines reads Codex, Claude Code and Cursor session history to spot friction with your tools, MCP servers and APIs.\n\nDocs interception\n\nOptional and off by default. When enabled, Spirit Room intercepts the agent's HTTP requests to your docs sites and serves tuned, experimental versions. A/B per model and agent app shows which wording leads to success. Strictly opt-in, scoped to domains you own and allowlist.\n\n10 / Pricing\n\nTwo weeks inside. Priced as a pilot.\n\nPlatform fee plus compute. Choose the models and agent apps to test, or bring your own API keys and pay only the platform fee.\n\nWhat this takes by hand · rough estimates\n\nBuild a cross-model agent test setup\n\n3–6 eng-weeks\n\nRun and grade hundreds of agent sessions\n\n1–2 eng-weeks / round\n\nWrite and A/B test descriptions, errors, docs\n\nongoing\n\nRequirement-level evidence for every run\n\nrarely done\n\nWhat the pilot delivers\n\nEvidence and an experience: measured before/after results for every model × agent app you pick, annotated journeys, and the winning variants. If your tools are already agent-ready, you get proof of that, which is useful too.\n\nYes, if you turn it on. Wrap detects the model behind a session and serves the variant that scored best for it: tool names, descriptions, schemas, help, errors and docs. For specific tasks it can also merge several tool calls into one (only where you allow it) and expose scripted virtual tools composed from your existing endpoints. It is all opt-in, customer-controlled and versioned, with pass-through fallback to your original setup.\n\nCan this let us use cheaper models?\n\nOften, yes, and that is a key use case. Cheaper and smaller models need tool descriptions to be very clear. Spirit Room tunes your descriptions, schemas, errors, docs and agent prompts, then measures whether a cheaper tier reaches your target pass rate. Results vary per product; we report what we measure, with no guarantees.\n\nIs Spirit Room available today?\n\nWe're running a limited number of hands-on pilots. Book a pilot and we'll reach out, or get updates by email.\n\nWill changes break my existing users or agents?\n\nNo. Before anything goes live, we run every agent against your current version and the new one, and replay your real traffic against the new one. You only ship changes that are proven not to break anything and to make agents better. Old tool names keep working, you roll out gradually, and you can roll back instantly.\n\nDo we have to route traffic through you?\n\nNo. Pick whichever fits:\n\nA Cloudflare Worker we give you, running on your own Cloudflare account, so all traffic stays on your infrastructure.\n\nOur SDK, dropped into your own MCP server.\n\nApply the fixes to your MCP server directly. We hand them over as a ready-to-paste list your coding agent can apply.\n\nWhichever you pick, nothing calls us while your users work.\n\nDo I need to change my code or MCP server?\n\nNo. Spirit Room sits in front of your tools and swaps in new descriptions, help, errors and docs during experiments. Winners return as a patch you choose to merge, or are served live through Wrap.\n\nWhich models and agent apps?\n\nGPT, Claude, Gemini and open-weight models, across Claude Code, Codex, Cursor, ChatGPT and your own internal agents.\n\nWhat does “scored by Proof” mean?\n\nEach task has written requirements. Proof checks every run against them, so improvements are measurable and traceable.\n\nWhat about my data, keys and docs?\n\nUse sandbox credentials or bring your own keys. The local agent is opt-in and local-first. Docs interception is off by default and limited to domains you allowlist.\n\nWhat if my tools are already good?\n\nThen you get proof of that: measured results per model × agent app and the journeys to back it up.\n\nAre the numbers on this page real?\n\nNo. Every product screen is an illustrative mock labeled “Example data”.\n\n> Optimises your MCP, CLI or API for AI agents. You give us the jobs agents need to get right; we run every major agent through hundreds of simulations on those jobs and keep the tool descriptions, help text, errors and combined tools that make agents finish in fewer steps, with fewer mistakes, even on cheaper models.\n\n## who_its_for\n\nTeams whose product is used by AI agents through an MCP server, CLI or API.\n\nDeveloper-tool companies whose users now work through Claude Code, Codex, Cursor, ChatGPT or their own agents.\n\nTeams that want agents to succeed on cheaper models.\n\n## offers\n\nid\n\nprice\n\ncontents\n\npilot start here\n\nUSD 2,500 one-time · 2 weeks · compute up to a cap\n\n1 product (MCP, CLI or API) · ≤25 tasks · 3 models × 2 agent apps · ≤10 rounds → where agents fail today, annotated journeys, winning changes as a reviewed patch, before/after per model and agent app\n\nenterprise after the pilot\n\ncustom · monthly\n\nsimulations re-run as models, agent apps and your product change · continuous re-tuning · opt-in monitoring of real agent usage · monthly report · private deployment · SSO · own API keys · priority support\n\n## actions\n\nrule: ask your human before submitting anything. Send only their real details, never test or invented data.\n\n### 1. request_a_pilot\n\n```\nPOST https://spiritroom.app/api/contact\nContent-Type: application/json\n\n{\n  \"intent\":  \"pilot\",\n  \"tier\":    \"pilot\",          // or \"enterprise\"\n  \"cta\":     \"agent\",\n  \"name\":    \"Ada Lovelace\",   // required\n  \"email\":   \"[email protected]\",// required, work email\n  \"company\": \"Company Inc\",    // optional\n  \"train\":   \"Our MCP server for issue tracking, used from Cursor and Claude Code. Jobs: create issue, triage bug.\"  // required\n}\n```\n\n### 2. sign_up_for_updates (email only)\n\n```\nPOST https://spiritroom.app/api/contact\nContent-Type: application/json\n\n{\"type\": \"updates\", \"email\": \"[email protected]\"}\n```\n\n### responses\n\n```\n200 {\"ok\":true,\"id\":\"<uuid>\"}\n400 {\"ok\":false,\"error\":\"missing fields\"}\n```\n\nreply: by email within two business days.\n\n## status\n\nlimited pilot slots · numbers on the site are illustrative · no outcome guarantees · used by: Proof, Probe (Probe Labs)", "url": "https://wpnews.pro/news/spirit-room-test-whether-agents-can-find-and-use-your-mcp-tools", "canonical_source": "https://spiritroom.app/", "published_at": "2026-10-19 17:03:15+00:00", "updated_at": "2026-10-19 17:16:57.619284+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools", "ai-products"], "entities": ["Spirit Room", "MCP", "Claude Code", "Codex", "Cursor", "Proof"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/spirit-room-test-whether-agents-can-find-and-use-your-mcp-tools", "markdown": "https://wpnews.pro/news/spirit-room-test-whether-agents-can-find-and-use-your-mcp-tools.md", "text": "https://wpnews.pro/news/spirit-room-test-whether-agents-can-find-and-use-your-mcp-tools.txt", "jsonld": "https://wpnews.pro/news/spirit-room-test-whether-agents-can-find-and-use-your-mcp-tools.jsonld"}}