My Demo Script Found a Production Bug on Its First Run: A Tiny Post-Mortem

A developer discovered a production bug in an MCP server for Google's Gemini model when a demo script failed on its first run. The server's local validation allowed four thinking levels, but the live API only accepts two, causing every default call to fail with a 400 error. The bug was invisible to a fully green mocked test suite, highlighting the risk of validating against a local copy of an API contract rather than the actual remote contract.

I built a demo script to show off a project. It failed on step 2 — and in doing so, it found a real bug that had been sitting in the codebase the whole time, invisible to a fully green test suite. This is a short technical report on what happened, why the tests missed it, and what it taught me about validating against remote APIs. Total incident cost: about ten minutes. Lessons: worth writing down. The project is an MCP Model Context Protocol server https://hub.docker.com/r/xbill9/nb2lite-mcp that wraps Google's gemini-3.1-flash-lite-image model — image generation and stateful editing exposed as four tools that any MCP client Claude Code, a Google ADK agent, a Rust CLI can call. I've written up the architecture separately https://dev.to/xbill/build-one-ai-tool-server-call-it-from-three-different-agents-mcp-explained-22l2 ; this post is only about the bug. The relevant detail: the server validates tool arguments locally before calling the API. One of those arguments is thinking level , the model's latency-vs-quality dial: server.py — as originally written SUPPORTED THINKING LEVELS = {"minimal", "low", "medium", "high"} @mcp.tool def generate image prompt: str, aspect ratio: str = "1:1", thinking level: str = "medium" - str: ... Four allowed values, defaulting to medium . The test suite — 10 unit tests, all mocked — passed. Lint passed. The server had been demoed through agents and worked. Ship it, right? I wrote a demo.sh that walks through the stack live: list the tools, generate an image, then do a stateful edit. To keep the demo cheap I picked the lowest thinking level: cargo run --quiet -- generate "a tiny robot chef cooking ramen" 16:9 minimal First run, step 2: 🔴 Image generation failed: Error code: 400 - {'error': {'message': "'minimal' is not a supported thinking level for this model. Allowed values are: low, high.", 'code': 'invalid request'}} The live API accepts exactly two thinking levels for this model: low and high . Not four. Read that against the code above and the real bug jumps out — and it's much worse than a demo flag being wrong: The server's defaultwas medium . Every live call that didn't explicitly override thinking level was a guaranteed HTTP 400. The local validation layer was happily approving values the API would reject, and rejecting nothing that mattered. It wasn't validating the contract; it was validating a memory of the contract. The test suite mocks the API client: python @patch "server. get client" def test generate image success self, mock get client : mock client.interactions.create.return value = mock interaction ... result = generate image prompt="test", thinking level="medium" self.assertIn "🟢 Image successfully saved ", result This is a good test — it verifies the server's own logic: argument handling, base64 decoding, file naming, error shaping. Mocked tests are supposed to isolate you from the network, and they do. But that isolation cuts both ways: a mocked test can never detect that the remote contract changed or was never what you thought . The mock returns whatever you tell it to, including for requests the real API would reject. My suite effectively asserted "the server correctly forwards medium to the API" — which it did. Correctly forwarding an invalid value is still a bug, just not one visible from inside the mock boundary. Two things had to line up for this to reach production: SUPPORTED THINKING LEVELS is a local copy of a fact the API owns. Local copies drift — whether because the docs were wrong, the model changed, or the set was written for a different model. high quality — masking the broken default and the two phantom values.Mechanically boring, which is the point — the hard part was knowing : -SUPPORTED THINKING LEVELS = {"minimal", "low", "medium", "high"} +SUPPORTED THINKING LEVELS = {"low", "high"} - prompt: str, aspect ratio: str = "1:1", thinking level: str = "medium" + prompt: str, aspect ratio: str = "1:1", thinking level: str = "low" Plus the sweep that actually takes the time: three tool signatures and docstrings, the server's self-describing get help text, the README, two articles, the API cheat-sheet doc — and a regression test that locks the new knowledge in : The live API only accepts low/high for this model; medium must be rejected result = generate image prompt="test", thinking level="medium" self.assertIn "Unsupported thinking level 'medium'", result Then verification against the real thing: a live call with pure default parameters — the exact case that was broken — returning 🟢 Image successfully saved . And because the server ships as a Docker image, a rebuild and push, since the published image contained the bug too. One design decision earned its keep during all this: the server's tools never raise — they catch everything and return human-readable 🔴 ... strings. The 400 came back as legible text an agent or a demo script, or me could read and act on, instead of a stack trace tearing down the MCP session. 1. Mocked tests verify your code. They cannot verify the contract. You need at least one test that touches the real API — even a single cheap smoke call. Mine now lives in the demo script and a /verify-stack routine: generate one tiny image with default parameters , because defaults are the values nobody passes explicitly and therefore nobody tests. 2. A local allowlist of remote-owned values is a drift time bomb. If you must pre-validate it does give agents faster, clearer errors than a round-trip 400 , treat the list as a cache of someone else's truth: comment where it came from, and pin a regression test to the values you've observed the API reject. 3. Test your defaults, specifically. The bug survived every live interaction before the demo because humans and agents kept overriding the broken default. f x being called a hundred times tells you nothing about f . 4. A demo script is the cheapest end-to-end test you'll ever write. It exercises the happy path a real user takes, with real credentials, against the real API — precisely the layer unit tests can't reach. Mine found a production bug on its first execution, before any audience did. Write the demo before you think you need one; run it in fast mode DEMO FAST=1 whenever the API-facing code changes. 5. Return errors agents can read. Tool calls that fail as readable text keep the conversation alive — the calling LLM can see Allowed values are: low, high and retry correctly on its own. That same property is what made this bug a ten-minute fix instead of a debugging session. | T+0 | demo.sh first run: step 2 fails with HTTP 400 | | T+1 min | Root cause identified from the error body: API allows low / high only | | T+4 min | Server validation + defaults fixed; regression test added | | T+6 min | Docs swept README, articles, cheat-sheet, skill ; 10/10 tests green | | T+8 min | Live verification with default params: 🟢 | | T+10 min | Fixed image rebuilt and pushed to Docker Hub | The demo, incidentally, works great now — the stateful edit produces the same cyberpunk kitchen with a new neon RAMEN sign, pixel-for-pixel continuity intact. But the bug report turned out to be the better story. Have your own "the demo found it" story? I'd love to hear it in the comments. 👇