{"slug": "show-hn-qamcp-automated-test-runner-for-mcp-servers-in-your-terminal", "title": "Show HN: QAmcp – Automated test runner for MCP servers in your terminal", "summary": "QAmcp released qamcp-cli, a Node.js 20+ command-line test runner that connects to Model Context Protocol (MCP) servers the way an AI app does and reports pass/fail results in plain language, with no AI judging and nothing sent anywhere except the server under test. In a demo against a deliberately broken local MCP server, QAmcp flagged hidden instructions that would cap the score at 40, a leaked database password in a stack trace, a 3-second slow tool, missing descriptions and hangs, while skipping data-changing tools such as delete_account and team_digest entirely. Testing Context7's public server (v4.4.1, MCP version 2026-07-28) with QAmcp 0.1.0 produced a score of 100/100 across 10 checks, with 852 ms to connect, 261 ms to list tools and 1.1 s average tool response.", "body_md": "**QAmcp tests MCP servers for the people who build them: it connects like an AI app does and tells you, in plain words, what's broken.**\n\nEvery result is a hard fact (pass or fail, milliseconds, exact matches). No AI judges anything, and nothing is sent anywhere except to the server you test.\n\n**Website:** [qamcp.github.io](https://qamcp.github.io): how it works, sample reports, and running it on every change.\n\nYou need [Node.js](https://nodejs.org) 20 or newer.\n\n```\nnpx qamcp-cli https://your-server.com/mcp              # test a server on the internet\nnpx qamcp-cli --stdio \"npx -y @your-org/your-server\"   # or a server that runs on your computer\nnpx qamcp-cli --help                                   # every option\n```\n\nOr install it once with `npm install -g qamcp-cli`, then just type `qamcp`.\n\n```\nnpx qamcp-cli --demo\n```\n\nThis starts an MCP server we broke on purpose, on your computer only, tests it, and writes `qamcp-demo-report.html`. It takes about 30 seconds. What the demo server hides, and what QAmcp says about it:\n\n| Tool | Planted problem | QAmcp finds | \n|---|---|---|\n| `fetch_notes` | A hidden instruction telling the AI to read your SSH key | ❌ Hidden instructions (on their own, they would cap the score at 40) | \n| `search_users` | A bad input shows a stack trace and a database password; slow (3 s); an input with no description | ❌ Secret leaked, | \n| `slow_report` | No description, and it never answers | ❌ No description, ❌ no answer, ❌ hangs on bad input | \n| `fetch_notes` | Accepts any input without complaint |  | \n| `strict_lookup` | Refuses bad input properly, but with HTTP 400 |  | \n| `delete_account` ,`team_digest` ,`get_or_create_user` | Change data, with no safety label | Not run, ever; | \n| `list_projects` | No label, but only reads | Not run in safe mode; runs with `--owner` | \n| `get_weather` | Nothing: the healthy control | ✅ Works | \n\nIf you find a broken server QAmcp misses, [tell us](https://github.com/QAmcp/qamcp-cli/issues).\n\nReal output from testing [Context7](https://context7.com)'s public server with QAmcp 0.1.0:\n\n```\nQAmcp report for https://mcp.context7.com/mcp\nServer: Context7 v4.4.1\nMCP version: 2026-07-28 (streamable-http)\n\nSCORE: 100/100\nTested 10 of 10 checks · ran 2 of 2 tools\nPrivacy: nothing was sent anywhere except to the server being tested (and its login server, if it needed a login). QAmcp has no tracking.\n\n✅ Connects — All good\n✅ Ready for the newest MCP version — All good\n✅ Lists its tools — All good\n✅ Every tool is described — All good\n✅ Tools are labeled safe or dangerous — All good\n✅ No hidden instructions — All good\n✅ Tools work with good inputs — All good\n✅ Bad inputs give a clean error — All good\n✅ Fast enough — All good\n✅ Connecting took 852 ms\n✅ Listing tools took 261 ms\n✅ Average tool answer: 1.1 s\n✅ No secrets leaked — All good\n✅ Checked 5 error messages: no secrets or internal details\n```\n\n| Check | What it looks at | \n|---|---|\n| **Connects** | Current and older MCP connection types, or a program on your computer (stdio) | \n| **Ready for the newest MCP version** | On the July 2026 version? Still on the old SSE connection? | \n| **Lists its tools** | Duplicate or invalid names, a broken tool that stops AI apps loading the server | \n| **Every tool is described** | Missing or vague descriptions, undescribed inputs, look-alike tools | \n| **Tools are labeled safe or dangerous** | Safety labels, and labels the tool itself contradicts | \n| **No hidden instructions** | Known tricks in descriptions: `<IMPORTANT>` blocks, \"don't tell the user\", invisible characters | \n| **Tools work with good inputs** | Runs safe tools; catches tools that hang or claim success but return nothing | \n| **Bad inputs give a clean error** | Missing and wrong-type inputs | \n| **Fast enough** | Connect, list and tool times | \n| **No secrets leaked** | Keys, passwords, server folders and stack traces in error messages | \n| **Login setup follows the spec** | Only when a server needs a login: the OAuth details AI apps need to log in | \n| **Resources and prompts work** | Only when a server has them: each opens, content matches its type, inputs are used | \n| **Tools work together** | Only when tools chain: one tool's answer (like an ID) is passed to the next tool, which must be able to use it | \n\nThe score is plain arithmetic, so you can check it yourself.\n\n1. \n**Each check is worth points.** A pass earns all of them, a warning half, a failure none.Check Points Connects, Every tool is described 15 each Newest MCP version, Lists its tools, Safety labels, No hidden instructions, Good inputs, Bad inputs 10 each Login setup, Resources and prompts, Tools work together (only when they apply) 10 each Fast enough, No secrets leaked 5 each\n2. \n**Checks that could not run don't count, either way.** The score is points earned out of the points that were actually tested, so a server isn't punished for something we couldn't try.\n3. \n**Serious problems put a ceiling on the score** , however good the rest is. The lowest ceiling wins:Problem Highest possible score Hidden instructions in a tool, resource or prompt description 40 A secret (key, password, private key) leaked in an error message 50 A broken tool definition that can stop AI apps loading the server 50 A tool that crashes or hangs, or claims success but returns nothing 70 When a ceiling applies, the report says so: *\"Would be 85, but capped at 50 because…\"*\n4. \n**Always read the score together with the coverage line,** e.g.`Tested 10 of 10 checks · ran 5 of 9 tools` . A 90 where only 2 of 20 tools could run says much less than a 90 where all 20 ran. Tools are only run when that's safe (see below), so coverage tells you how much the score really covers.\n5. \n**\"NOT SCORED\"** means we couldn't test the server for a reason that isn't its fault, like a login wall. Add a key with`--header` or log in with`--login` , then test again.\n\nRough guide: **90+** solid, **70–89** works but has things to fix, **below 70** AI apps will likely struggle, or there is a safety problem.\n\n- **It never runs a tool that could change something.** By default it runs only tools the server labels read-only.\n- **It never runs a tool that sounds destructive** , like`delete_account` , in any mode.\n- **It never sends your data anywhere else.** No AI, no tracking, no telemetry.\n- **For a program on your computer (`--stdio`), it starts the program a second time for a moment** to check that it prints only MCP messages, and says so in the report.\n\nIf you own the server, `--owner` also runs unlabeled tools whose names clearly only read (`get_`, `list_`, `search_`…). Tools whose name or description says they write, send or create are still skipped.\n\nWe keep a test server with one planted problem per tool ([`test/buggyServer.ts`](https://github.com/QAmcp/qamcp-cli/blob/main/test/buggyServer.ts)). Real output from QAmcp 0.1.0, shortened:\n\n```\nSCORE: 38/100\nTested 10 of 10 checks · ran 5 of 9 tools\n\n❌ slow_report: Has no description, so the AI has to guess what it does\n❌ fetch_notes: Description contains hidden <IMPORTANT> block. This is a known trick to make AI do things the user didn't ask for.\n❌ slow_report: No answer within the time limit\n❌ slow_report: Hung on missing required input \"id\"\n❌ search_users: Error message shows what looks like a database password in a link\n⚠️  search_users: Error message shows internal details: code stack trace, server file path\n⚠️  fetch_notes: Accepted missing required input \"key\" without complaining\n⚠️  strict_lookup: Errors are sent with an HTTP error status (like 400) instead of a normal MCP reply\n⚠️  delete_account: Name sounds like it deletes something, but it is not labeled dangerous\n➖ delete_account: Not run: name sounds like it deletes something\n```\n\nWith `--owner`, it also ran `list_projects` (unlabeled, name only reads) and still skipped `team_digest` (\"Sends\" in its description), `get_or_create_user` (\"create\" in its name) and `delete_account`.\n\n```\nnpx qamcp-cli https://your-server.com/mcp --header \"Authorization: Bearer YOUR_KEY\"   # a server with a key\nnpx qamcp-cli https://your-server.com/mcp --login                                     # log in through your browser\nnpx qamcp-cli https://your-server.com/mcp --json report.json                         # save the full report\nnpx qamcp-cli https://your-server.com/mcp --html report.html                         # a report page you can open and share (works offline)\nnpx qamcp-cli https://your-server.com/mcp --look-only                              # call no tools at all: a safe first look at a server you don't trust\n```\n\nAdd `.github/workflows/qamcp.yml` to your repository:\n\n```\nname: QAmcp\non: [pull_request]\npermissions:\n  contents: read\n  pull-requests: write\njobs:\n  qamcp:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v7\n      - uses: QAmcp/qamcp-cli@v0\n        with:\n          url: https://your-server.com/mcp\n          fail-under: 80\n```\n\nEvery pull request gets a comment with the score, and the build fails if the server got worse. [`examples/qamcp-ci.yml`](https://github.com/QAmcp/qamcp-cli/blob/main/examples/qamcp-ci.yml) shows how to compare with the last run and catch tools that changed or disappeared.\n\nAny other CI system:\n\n```\nnpx qamcp-cli https://your-server.com/mcp --fail-under 80 --baseline base.json --save-baseline base.json --junit report.xml\n```\n\nExit codes: `0` fine · `1` a check you required failed · `2` wrong options · `3` could not test (server down, or needs a login).\n\n```\ndocker build -t qamcp .\ndocker run --rm qamcp https://your-server.com/mcp\njs\nimport { runTest } from 'qamcp-cli';\n\nconst report = await runTest('https://your-server.com/mcp');\nconsole.log(report.score, report.checks);\n```\n\n[Open an issue](https://github.com/QAmcp/qamcp-cli/issues). If QAmcp gets something wrong about your server, please tell us: being fair matters more to us than anything else.\n\nMIT", "url": "https://wpnews.pro/news/show-hn-qamcp-automated-test-runner-for-mcp-servers-in-your-terminal", "canonical_source": "https://github.com/QAmcp/qamcp-cli", "published_at": "2026-10-21 20:14:37+00:00", "updated_at": "2026-10-21 20:42:19.907778+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "developer-tools", "ai-tools", "ai-safety"], "entities": ["QAmcp", "qamcp-cli", "Model Context Protocol", "Context7", "Node.js", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-qamcp-automated-test-runner-for-mcp-servers-in-your-terminal", "markdown": "https://wpnews.pro/news/show-hn-qamcp-automated-test-runner-for-mcp-servers-in-your-terminal.md", "text": "https://wpnews.pro/news/show-hn-qamcp-automated-test-runner-for-mcp-servers-in-your-terminal.txt", "jsonld": "https://wpnews.pro/news/show-hn-qamcp-automated-test-runner-for-mcp-servers-in-your-terminal.jsonld"}}