{"slug": "my-demo-script-found-a-production-bug-on-its-first-run-a-tiny-post-mortem", "title": "My Demo Script Found a Production Bug on Its First Run: A Tiny Post-Mortem", "summary": "A developer discovered a production bug in an MCP server for Google's Gemini model when a demo script failed on its first run. The server's local validation allowed four thinking levels, but the live API only accepts two, causing every default call to fail with a 400 error. The bug was invisible to a fully green mocked test suite, highlighting the risk of validating against a local copy of an API contract rather than the actual remote contract.", "body_md": "I built a demo script to show off a project. It failed on step 2 — and in doing so, it found a real bug that had been sitting in the codebase the whole time, invisible to a fully green test suite.\n\nThis is a short technical report on what happened, why the tests missed it, and what it taught me about validating against remote APIs. Total incident cost: about ten minutes. Lessons: worth writing down.\n\nThe project is an [MCP (Model Context Protocol) server](https://hub.docker.com/r/xbill9/nb2lite-mcp) that wraps Google's `gemini-3.1-flash-lite-image`\n\nmodel — image generation and stateful editing exposed as four tools that any MCP client (Claude Code, a Google ADK agent, a Rust CLI) can call. I've written up [the architecture separately](https://dev.to/xbill/build-one-ai-tool-server-call-it-from-three-different-agents-mcp-explained-22l2); this post is only about the bug.\n\nThe relevant detail: the server validates tool arguments *locally* before calling the API. One of those arguments is `thinking_level`\n\n, the model's latency-vs-quality dial:\n\n```\n# server.py — as originally written\nSUPPORTED_THINKING_LEVELS = {\"minimal\", \"low\", \"medium\", \"high\"}\n\n@mcp.tool()\ndef generate_image(\n    prompt: str, aspect_ratio: str = \"1:1\", thinking_level: str = \"medium\"\n) -> str:\n    ...\n```\n\nFour allowed values, defaulting to `medium`\n\n. The test suite — 10 unit tests, all mocked — passed. Lint passed. The server had been demoed through agents and worked. Ship it, right?\n\nI wrote a `demo.sh`\n\nthat walks through the stack live: list the tools, generate an image, then do a stateful edit. To keep the demo cheap I picked the *lowest* thinking level:\n\n```\ncargo run --quiet -- generate \"a tiny robot chef cooking ramen\" 16:9 minimal\n```\n\nFirst run, step 2:\n\n```\n🔴 Image generation failed: Error code: 400 - {'error': {'message':\n\"'minimal' is not a supported thinking level for this model.\nAllowed values are: low, high.\", 'code': 'invalid_request'}}\n```\n\nThe live API accepts exactly **two** thinking levels for this model: `low`\n\nand `high`\n\n. Not four.\n\nRead that against the code above and the real bug jumps out — and it's much worse than a demo flag being wrong:\n\nThe server's\n\ndefaultwas`medium`\n\n. Every live call that didn't explicitly override`thinking_level`\n\nwas a guaranteed HTTP 400.\n\nThe local validation layer was happily approving values the API would reject, and rejecting nothing that mattered. It wasn't validating the contract; it was validating a *memory* of the contract.\n\nThe test suite mocks the API client:\n\n``` python\n@patch(\"server._get_client\")\ndef test_generate_image_success(self, mock_get_client):\n    mock_client.interactions.create.return_value = mock_interaction\n    ...\n    result = generate_image(prompt=\"test\", thinking_level=\"medium\")\n    self.assertIn(\"🟢 Image successfully saved!\", result)\n```\n\nThis is a *good* test — it verifies the server's own logic: argument handling, base64 decoding, file naming, error shaping. Mocked tests are supposed to isolate you from the network, and they do.\n\nBut that isolation cuts both ways: **a mocked test can never detect that the remote contract changed** (or was never what you thought). The mock returns whatever you tell it to, including for requests the real API would reject. My suite effectively asserted \"the server correctly forwards `medium`\n\nto the API\" — which it did. Correctly forwarding an invalid value is still a bug, just not one visible from inside the mock boundary.\n\nTwo things had to line up for this to reach production:\n\n`SUPPORTED_THINKING_LEVELS`\n\nis a local copy of a fact the API owns. Local copies drift — whether because the docs were wrong, the model changed, or the set was written for a different model.`high`\n\n(quality) — masking the broken default and the two phantom values.Mechanically boring, which is the point — the hard part was *knowing*:\n\n```\n-SUPPORTED_THINKING_LEVELS = {\"minimal\", \"low\", \"medium\", \"high\"}\n+SUPPORTED_THINKING_LEVELS = {\"low\", \"high\"}\n\n-    prompt: str, aspect_ratio: str = \"1:1\", thinking_level: str = \"medium\"\n+    prompt: str, aspect_ratio: str = \"1:1\", thinking_level: str = \"low\"\n```\n\nPlus the sweep that actually takes the time: three tool signatures and docstrings, the server's self-describing `get_help`\n\ntext, the README, two articles, the API cheat-sheet doc — and a **regression test that locks the new knowledge in**:\n\n```\n# The live API only accepts low/high for this model; medium must be rejected\nresult = generate_image(prompt=\"test\", thinking_level=\"medium\")\nself.assertIn(\"Unsupported thinking level 'medium'\", result)\n```\n\nThen verification against the real thing: a live call with pure default parameters — the exact case that was broken — returning `🟢 Image successfully saved!`\n\n. And because the server ships as a Docker image, a rebuild and push, since the published image contained the bug too.\n\nOne design decision earned its keep during all this: the server's tools never raise — they catch everything and return human-readable `🔴 ...`\n\nstrings. The 400 came back as legible text an agent (or a demo script, or me) could read and act on, instead of a stack trace tearing down the MCP session.\n\n**1. Mocked tests verify your code. They cannot verify the contract.** You need at least one test that touches the real API — even a single cheap smoke call. Mine now lives in the demo script and a `/verify-stack`\n\nroutine: generate one tiny image with *default parameters*, because defaults are the values nobody passes explicitly and therefore nobody tests.\n\n**2. A local allowlist of remote-owned values is a drift time bomb.** If you must pre-validate (it does give agents faster, clearer errors than a round-trip 400), treat the list as a cache of someone else's truth: comment where it came from, and pin a regression test to the values you've *observed* the API reject.\n\n**3. Test your defaults, specifically.** The bug survived every live interaction before the demo because humans and agents kept overriding the broken default. `f(x)`\n\nbeing called a hundred times tells you nothing about `f()`\n\n.\n\n**4. A demo script is the cheapest end-to-end test you'll ever write.** It exercises the happy path a real user takes, with real credentials, against the real API — precisely the layer unit tests can't reach. Mine found a production bug on its first execution, before any audience did. Write the demo *before* you think you need one; run it in fast mode (`DEMO_FAST=1`\n\n) whenever the API-facing code changes.\n\n**5. Return errors agents can read.** Tool calls that fail as readable text keep the conversation alive — the calling LLM can see `Allowed values are: low, high`\n\nand retry correctly on its own. That same property is what made this bug a ten-minute fix instead of a debugging session.\n\n| T+0 |\n`demo.sh` first run: step 2 fails with HTTP 400 |\n| T+1 min | Root cause identified from the error body: API allows `low` /`high` only |\n| T+4 min | Server validation + defaults fixed; regression test added |\n| T+6 min | Docs swept (README, articles, cheat-sheet, skill); 10/10 tests green |\n| T+8 min | Live verification with default params: 🟢 |\n| T+10 min | Fixed image rebuilt and pushed to Docker Hub |\n\nThe demo, incidentally, works great now — the stateful edit produces the same cyberpunk kitchen with a new neon RAMEN sign, pixel-for-pixel continuity intact. But the bug report turned out to be the better story.\n\n*Have your own \"the demo found it\" story? I'd love to hear it in the comments.* 👇", "url": "https://wpnews.pro/news/my-demo-script-found-a-production-bug-on-its-first-run-a-tiny-post-mortem", "canonical_source": "https://dev.to/xbill/my-demo-script-found-a-production-bug-on-its-first-run-a-tiny-post-mortem-k35", "published_at": "2026-07-15 20:36:09+00:00", "updated_at": "2026-07-15 21:10:22.454593+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "large-language-models"], "entities": ["Google", "Gemini", "MCP", "Claude Code", "Rust"], "alternates": {"html": "https://wpnews.pro/news/my-demo-script-found-a-production-bug-on-its-first-run-a-tiny-post-mortem", "markdown": "https://wpnews.pro/news/my-demo-script-found-a-production-bug-on-its-first-run-a-tiny-post-mortem.md", "text": "https://wpnews.pro/news/my-demo-script-found-a-production-bug-on-its-first-run-a-tiny-post-mortem.txt", "jsonld": "https://wpnews.pro/news/my-demo-script-found-a-production-bug-on-its-first-run-a-tiny-post-mortem.jsonld"}}