{"slug": "model-experiments-became-an-architectural-stress-test", "title": "Model experiments became an architectural stress test", "summary": "A developer tuning Codenames AI, a web game where an LLM plays Codenames with users, found that switching the model from gpt-4o-mini to gpt-5-mini with minimal reasoning caused structural failures in validation layers that had been stable for months. The failures exposed gaps in contracts for clue generation, including side commentary, target cardinality mismatches, and underspecified retry logic, turning model experiments into an architectural stress test.", "body_md": "I've been tuning [Codenames AI](https://codenames-ai.com/?utm_source=devto&utm_medium=article&utm_campaign=model-experiments-became-an-architectural-stress-test&utm_content=intro), a small web game where an LLM plays Codenames with you. Clue generation is tightly constrained: one word, a count, optional intended targets, JSON on the wire, then deterministic validation before anything reaches the board.\n\nAs the project started attracting regular players, I wanted to improve the gameplay experience without blowing out costs. Moving one model generation from `gpt-4o-mini`\n\nto `gpt-5-mini`\n\nwas my first instinct.\n\nThe default reasoning setting made responses an order of magnitude slower for this workload. Minimal reasoning looked like the obvious compromise: newer model, responsive gameplay.\n\nI expected to compare clue quality, latency, and cost while the surrounding prompt, validator, and consumer contracts stayed put.\n\nThat last part was wrong.\n\nWhat showed up was structural, and it showed up in places that had been stable for months.\n\nValidation failures started rising. Retries started rising. Entire candidate batches started failing before the game ever saw a clue. The sharpest signal came from a clue-selection path that had run untouched for months, and it hard-failed for the first time. They weren't latency regressions so much as architectural ones.\n\nIt is easy to read that as \"minimal reasoning made the model worse.\" More often, the failures were exposing gaps in contracts that had looked fine under the previous model.\n\nEventually every failure traced back to one of three layers:\n\n`count`\n\ntargets and, in batch mode, several distinct candidates.Those layers share one job: enforce the same invariants. The failures below cut across all three rather than mapping one to one.\n\n**Side commentary could kill an otherwise usable turn.**\n\nTo pick a clue, one strategy (Strange mode) simulates how the AI guesser would respond to each candidate clue, then scores those simulated turns and keeps the best one. I thought those simulations would fail only when the guesses themselves were bad. After the swap, they could also fail because the model attached commentary about other words it had considered, including words that were not even on the board. Because the payload schema included that commentary, the validator had to treat it as part of the same all-or-nothing contract. A payload with usable guesses still got rejected, and when every candidate died that way, the turn came back as a controlled API failure instead of a clue.\n\n**Target cardinality had to match the clue count.**\n\nI thought my validator was protecting the game. Instead I discovered the previous model had been consistently producing outputs that satisfied those contracts.\n\nSay the prompt asks for `count: 2`\n\nand a `targets`\n\narray with exactly two unrevealed friendly codenames. Under the old model, a clue like `{\"word\": \"BUILDING\", \"count\": 2, \"targets\": [\"TOWER\", \"CASTLE\"]}`\n\nusually meant two real board words. After the swap, I started seeing the same shape with one valid target and one word that is not on the grid at all, or only a single target when `count`\n\nwas 2. Valid JSON. Perfect keys. Intent status: invalid.\n\nThe validator rejects clues whose validated targets don't match `count`\n\n. Valid JSON wasn't enough.\n\n**Retries assumed the contracts were already specific enough.**\n\nI thought retries were simply robustness. Instead they became diagnostic tooling because they finally told me which invariant had actually failed. When a batch fails validation, the retry path can attach rejection feedback (failed clue words plus reason strings) so the next attempt is not a blind redo. That only helps if the contracts are specific enough to name the failure. Vague \"try again\" prompts hide whether you have a model problem or an underspecified invariant.\n\n**The failures showed up in the product, not just the logs.**\n\nEvery rejected clue meant another retry before the player saw a move. On an AI spymaster turn, the game shows a clue, a count, and highlights the board words that clue is meant to cover. When the validated targets came back shorter than `count`\n\n, the UI looked broken: `count: 2`\n\nwith only one word highlighted. The AI guesser still trusted the clue count and started reasoning from a board state that never actually existed.\n\nNone of this required a different product thesis from [schema-first validation](https://dev.to/michaeltruong/schema-first-prompt-second-valid-json-wasnt-enough-3nhm). Valid JSON was never enough. The migration stress-tested whether prompt text, deterministic checks, and consumer assumptions still agreed after the model changed.\n\nOn paper, the clue path already looked responsible. Prompt, validator, consumer. Clean separation.\n\nThe migration revealed a hidden layer:\n\n```\nPrompt\n  ↓\nModel capability\n  (compensating for weak contracts)\n  ↓\nValidator\n  ↓\nConsumer\n```\n\nI expected to compare models. Instead I ended up comparing how much of my architecture each model had been compensating for.\n\nWhile a more capable model kept quietly covering those weak contracts, the dashboards looked fine. Drop reasoning effort, and the same prompts start producing outputs that are honest about what you actually specified. Once that stopped happening, I was no longer measuring model quality. I was measuring how much of the gameplay experience had been resting on those hidden assumptions.\n\nThat is uncomfortable and useful. Apparent regressions (count mismatches, partial batches, more retries, collapsed guess simulations) are a signal to ask which layer was doing the work: the model, or the application.\n\nSubjective \"does this clue feel clever?\" still matters for gameplay. It should not be the only scoreboard when the pipeline can reject an entire batch before the server ever picks a clue.\n\nWhat I want out of a model swap now:\n\n`count`\n\ntargets,\" the validator must reject mismatches, and the API response shape must not pretend invalid intent is OK.**Takeaway:** A model migration tests the model and the architecture around it. If prompt, validator, and consumer contracts do not enforce the same invariants, stronger models can mask weaknesses in those contracts until a cheaper or more literal model exposes them. The lesson is not really about which LLM you pick. It is about architectural coupling: the model itself had become part of the contract without me noticing.\n\nIf you'd like to see the project that inspired these lessons, you can try [Codenames AI](https://codenames-ai.com/?utm_source=devto&utm_medium=article&utm_campaign=model-experiments-became-an-architectural-stress-test&utm_content=footer).", "url": "https://wpnews.pro/news/model-experiments-became-an-architectural-stress-test", "canonical_source": "https://dev.to/michaeltruong/model-experiments-became-an-architectural-stress-test-3gc0", "published_at": "2026-07-17 15:41:21+00:00", "updated_at": "2026-07-17 15:59:41.425205+00:00", "lang": "en", "topics": ["large-language-models", "ai-products", "developer-tools"], "entities": ["Codenames AI", "gpt-4o-mini", "gpt-5-mini"], "alternates": {"html": "https://wpnews.pro/news/model-experiments-became-an-architectural-stress-test", "markdown": "https://wpnews.pro/news/model-experiments-became-an-architectural-stress-test.md", "text": "https://wpnews.pro/news/model-experiments-became-an-architectural-stress-test.txt", "jsonld": "https://wpnews.pro/news/model-experiments-became-an-architectural-stress-test.jsonld"}}