I built a demo script to show off a project. It failed on step 2 — and in doing so, it found a real bug that had been sitting in the codebase the whole time, invisible to a fully green test suite.
This is a short technical report on what happened, why the tests missed it, and what it taught me about validating against remote APIs. Total incident cost: about ten minutes. Lessons: worth writing down.
The project is an MCP (Model Context Protocol) server that wraps Google's gemini-3.1-flash-lite-image
model — image generation and stateful editing exposed as four tools that any MCP client (Claude Code, a Google ADK agent, a Rust CLI) can call. I've written up the architecture separately; this post is only about the bug.
The relevant detail: the server validates tool arguments locally before calling the API. One of those arguments is thinking_level
, the model's latency-vs-quality dial:
SUPPORTED_THINKING_LEVELS = {"minimal", "low", "medium", "high"}
@mcp.tool()
def generate_image(
prompt: str, aspect_ratio: str = "1:1", thinking_level: str = "medium"
) -> str:
...
Four allowed values, defaulting to medium
. The test suite — 10 unit tests, all mocked — passed. Lint passed. The server had been demoed through agents and worked. Ship it, right?
I wrote a demo.sh
that walks through the stack live: list the tools, generate an image, then do a stateful edit. To keep the demo cheap I picked the lowest thinking level:
cargo run --quiet -- generate "a tiny robot chef cooking ramen" 16:9 minimal
First run, step 2:
🔴 Image generation failed: Error code: 400 - {'error': {'message':
"'minimal' is not a supported thinking level for this model.
Allowed values are: low, high.", 'code': 'invalid_request'}}
The live API accepts exactly two thinking levels for this model: low
and high
. Not four.
Read that against the code above and the real bug jumps out — and it's much worse than a demo flag being wrong:
The server's
defaultwasmedium
. Every live call that didn't explicitly overridethinking_level
was a guaranteed HTTP 400.
The local validation layer was happily approving values the API would reject, and rejecting nothing that mattered. It wasn't validating the contract; it was validating a memory of the contract.
The test suite mocks the API client:
@patch("server._get_client")
def test_generate_image_success(self, mock_get_client):
mock_client.interactions.create.return_value = mock_interaction
...
result = generate_image(prompt="test", thinking_level="medium")
self.assertIn("🟢 Image successfully saved!", result)
This is a good test — it verifies the server's own logic: argument handling, base64 decoding, file naming, error shaping. Mocked tests are supposed to isolate you from the network, and they do.
But that isolation cuts both ways: a mocked test can never detect that the remote contract changed (or was never what you thought). The mock returns whatever you tell it to, including for requests the real API would reject. My suite effectively asserted "the server correctly forwards medium
to the API" — which it did. Correctly forwarding an invalid value is still a bug, just not one visible from inside the mock boundary.
Two things had to line up for this to reach production:
SUPPORTED_THINKING_LEVELS
is a local copy of a fact the API owns. Local copies drift — whether because the docs were wrong, the model changed, or the set was written for a different model.high
(quality) — masking the broken default and the two phantom values.Mechanically boring, which is the point — the hard part was knowing:
-SUPPORTED_THINKING_LEVELS = {"minimal", "low", "medium", "high"}
+SUPPORTED_THINKING_LEVELS = {"low", "high"}
- prompt: str, aspect_ratio: str = "1:1", thinking_level: str = "medium"
+ prompt: str, aspect_ratio: str = "1:1", thinking_level: str = "low"
Plus the sweep that actually takes the time: three tool signatures and docstrings, the server's self-describing get_help
text, the README, two articles, the API cheat-sheet doc — and a regression test that locks the new knowledge in:
result = generate_image(prompt="test", thinking_level="medium")
self.assertIn("Unsupported thinking level 'medium'", result)
Then verification against the real thing: a live call with pure default parameters — the exact case that was broken — returning 🟢 Image successfully saved!
. And because the server ships as a Docker image, a rebuild and push, since the published image contained the bug too.
One design decision earned its keep during all this: the server's tools never raise — they catch everything and return human-readable 🔴 ...
strings. The 400 came back as legible text an agent (or a demo script, or me) could read and act on, instead of a stack trace tearing down the MCP session.
1. Mocked tests verify your code. They cannot verify the contract. You need at least one test that touches the real API — even a single cheap smoke call. Mine now lives in the demo script and a /verify-stack
routine: generate one tiny image with default parameters, because defaults are the values nobody passes explicitly and therefore nobody tests.
2. A local allowlist of remote-owned values is a drift time bomb. If you must pre-validate (it does give agents faster, clearer errors than a round-trip 400), treat the list as a cache of someone else's truth: comment where it came from, and pin a regression test to the values you've observed the API reject.
3. Test your defaults, specifically. The bug survived every live interaction before the demo because humans and agents kept overriding the broken default. f(x)
being called a hundred times tells you nothing about f()
.
4. A demo script is the cheapest end-to-end test you'll ever write. It exercises the happy path a real user takes, with real credentials, against the real API — precisely the layer unit tests can't reach. Mine found a production bug on its first execution, before any audience did. Write the demo before you think you need one; run it in fast mode (DEMO_FAST=1
) whenever the API-facing code changes.
5. Return errors agents can read. Tool calls that fail as readable text keep the conversation alive — the calling LLM can see Allowed values are: low, high
and retry correctly on its own. That same property is what made this bug a ten-minute fix instead of a debugging session.
| T+0 |
demo.sh first run: step 2 fails with HTTP 400 |
| T+1 min | Root cause identified from the error body: API allows low /high only |
| T+4 min | Server validation + defaults fixed; regression test added |
| T+6 min | Docs swept (README, articles, cheat-sheet, skill); 10/10 tests green |
| T+8 min | Live verification with default params: 🟢 |
| T+10 min | Fixed image rebuilt and pushed to Docker Hub |
The demo, incidentally, works great now — the stateful edit produces the same cyberpunk kitchen with a new neon RAMEN sign, pixel-for-pixel continuity intact. But the bug report turned out to be the better story.
Have your own "the demo found it" story? I'd love to hear it in the comments. 👇