{"slug": "from-9-to-63-letting-a-benchmark-build-a-better-mcp-security-proxy", "title": "From 9% to 63%: Letting a Benchmark Build a Better MCP Security Proxy", "summary": "A developer built mcp-defense-bench, an open benchmark for evaluating security proxies for the Model Context Protocol (MCP), which allows AI assistants to interact with external tools. Starting from a 9% coverage baseline, the developer iteratively improved the mcp-bastion proxy to 63% coverage by addressing 24 attack vectors, including novel attacks like ShareLock that no other tool detected. The work demonstrates a measurement-driven approach to hardening AI-agent security infrastructure.", "body_md": "A year ago, \"AI security\" mostly meant \"don't let the chatbot say something dumb.\" Today, AI assistants\n\n*act*: they read your files, query your databases, hit APIs, and browse the web. They do it through a\n\nfast-spreading open standard called the **Model Context Protocol (MCP)** — think of it as USB for AI\n\nagents, one standard plug between an assistant and the tools it uses.\n\nThat plug is powerful, and it's a brand-new place for things to go wrong. A malicious or compromised\n\ntool can hide instructions in its own description, quietly swap its behavior after you've approved it,\n\nor trick the agent into shipping your credentials to an attacker. The 2025–2026 research literature now\n\ndocuments dozens of these attacks.\n\nSo the obvious question: **when you put a security tool in front of your MCP servers, how much of that\ndanger does it actually stop?** Nobody had a good number. I set out to get one — and then something more\n\n`mcp-bastion`\n\nThis is the story of that loop, and what it teaches about where AI-agent security actually stands.\n\nI built [ mcp-defense-bench](https://github.com/Gowthaman90/mcp-defense-bench), an open, vendor-neutral\n\nThe rubric behind it is a **threat–control crosswalk**: 24 MCP attack vectors, each mapped to an\n\narchitectural layer, a STRIDE class, and the governance frameworks organizations actually use — the\n\nNIST AI Risk Management Framework, the NSA's MCP Security guidance, and the OWASP Top 10s for LLM and\n\nAgentic applications. (If you want the \"map of the whole landscape\" version of this story, that's a\n\na separate write-up; here I'm focused on the improvement loop.)\n\nEach attack scores one of three levels: **none** (missed), **detect** (warns you), or **enforce**\n\n(blocks it). Weighted coverage counts an enforce as 1.0, a detect as 0.5.\n\nHere's the part that surprised me. A benchmark is usually a scoreboard you check at the end. This one\n\nbecame a to-do list.\n\nThe proxy started at **9%** — it scanned tool *descriptions* for poisoning and nothing else. Then the\n\nbenchmark pointed at the next gap, I closed it, and re-ran. Every step below was re-verified at **zero\nfalse positives**:\n\n| Step | What the benchmark exposed → what I added | Coverage |\n|---|---|---|\n| baseline | description scanning only | 9% |\n| 1 | result-borne injection → scan tool results, not just definitions |\n18% |\n| 2 | parameter smuggling → validate arguments against the tool's schema | 23% |\n| 3 | DNS-rebinding on local servers → an origin check that blocks\n|\n30% |\n| 4 | cross-tool exfiltration → sensitive-argument + least-privilege scanning | 34% |\n| 5 | a brand-new split-payload attack → cross-tool correlation | 38% |\n| 6 | four vectors no tool covered → command-injection, config-drift, server-identity, cross-server taint |\n48% |\n| 7 | depth, not breadth → evasion normalization + inline secret redaction + block-by-default | 63% |\n\nTwo moments in that table are worth slowing down on, because they're where \"measure a defense\" turned\n\ninto \"discover something.\"\n\nHalfway through, two attacks were published (June 2026) that my frozen rubric didn't have yet:\n\n**mid-session tool injection** (a tool that mutates its behavior *after* you've approved it) and\n\n**ShareLock** (a stealthy attack that splits a malicious payload across *several* tools, so no single\n\ntool description looks suspicious).\n\nI verified both against their source papers, encoded each as a matched attack/benign pair, and re-ran\n\nevery tool. The result was stark: **ShareLock was covered by nothing.** Every scanner looked at one tool\n\nat a time and saw nothing wrong.\n\nBut a *proxy* sees the whole tool set at once. So I added **cross-tool correlation** — it scans the\n\ncombined descriptions and flags the coordinated \"share/checksum\"-style staging metadata the attack uses.\n\n`mcp-bastion`\n\nbecame the first and (on the benchmark) only measured tool to catch it. A benchmark that\n\nonly tested a fixed set of attacks would have missed this entirely. The value was in *re-measuring as\nthe threat surface moved.*\n\nLater, I added **cross-server data-flow tracking**. Picture an agent reading a secret from your vault\n\nserver and then, two calls later, posting it to a webhook on a *different* server. No single tool did\n\nanything wrong; the danger is in the *flow across a trust boundary*. A per-server scanner is blind to\n\nit. A proxy that fronts all your servers isn't — it can watch a credential-shaped token read from server\n\nA reappear in an argument headed to server B, and stop it. That's not a rule you can bolt onto one\n\ntool; it's a capability you only get from the vantage point.\n\nA tool that *detects* an attack and prints a warning is a linter. A tool that *blocks* it is a firewall.\n\nThe last leg of the journey (48% → 63%) was mostly about turning warnings into blocks — safely.\n\nThrough all of it, the false-positive count stayed at **zero across 35 test cases**. That's the whole\n\npoint of the matched benign controls: you can push toward \"block by default\" *because* you've proven the\n\nchecks don't fire on innocent traffic.\n\nHere's the result I care about most, and it's not the 63%. It's the **5 of 24 vectors that no measured\ntool covers — and that no content-scanning proxy ever will:**\n\nA proxy is *necessary but not sufficient.* The data makes the case for **defense-in-depth across\ndistinct mechanism classes** better than any argument could: 19 of 24 vectors are caught by\n\nThe loop is no longer manual. `mcp-bastion`\n\n's continuous-integration pipeline runs the full benchmark\n\non **every change** and fails the build if coverage drops below a floor or any false positive appears.\n\nThe measurement isn't a one-time paper result; it's a standing regression test on the defense. When the\n\nnext MCP attack drops, the process for handling it is already wired: add the vector, re-measure, close\n\nthe gap, and the gate makes sure it stays closed.\n\nEverything is open:\n\n`mcp-defense-bench`\n\n`mcp-bastion`\n\n`npx -y mcp-bastion`\n\n) and in the official MCP registry.One disclosure, up front: **I build one of the tools the benchmark measures.** That's exactly why the\n\nwhole thing is open, vendor-neutral, driven through each tool's real code, and scored alongside\n\ncompetitors and a do-nothing baseline — so you don't have to take my word for any of it. Re-run it\n\nyourself.\n\nThe number that matters isn't 63%. It's that an honest measurement, applied relentlessly, turned a\n\n9%-coverage scanner into the broadest MCP defense measured — and drew a bright line around the 37% that\n\nneeds a fundamentally different kind of tool. That line is the useful part.\n\n*If you work on MCP, agent security, or AI governance, I'd genuinely value critique of the rubric and\nthe methodology — open an issue, or reach me at agowthaman90@gmail.com.*", "url": "https://wpnews.pro/news/from-9-to-63-letting-a-benchmark-build-a-better-mcp-security-proxy", "canonical_source": "https://dev.to/gowthaman90/from-9-to-63-letting-a-benchmark-build-a-better-mcp-security-proxy-3gg8", "published_at": "2026-07-16 21:09:21+00:00", "updated_at": "2026-07-16 21:36:24.786888+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-infrastructure", "ai-research", "developer-tools"], "entities": ["mcp-defense-bench", "mcp-bastion", "Model Context Protocol", "NIST AI Risk Management Framework", "OWASP", "NSA"], "alternates": {"html": "https://wpnews.pro/news/from-9-to-63-letting-a-benchmark-build-a-better-mcp-security-proxy", "markdown": "https://wpnews.pro/news/from-9-to-63-letting-a-benchmark-build-a-better-mcp-security-proxy.md", "text": "https://wpnews.pro/news/from-9-to-63-letting-a-benchmark-build-a-better-mcp-security-proxy.txt", "jsonld": "https://wpnews.pro/news/from-9-to-63-letting-a-benchmark-build-a-better-mcp-security-proxy.jsonld"}}