# The Eval Flywheel: Turning Every Production AI Failure Into a Regression Test > Source: > Published: 2026-07-16 12:02:21+00:00 Member-only story # The Eval Flywheel: Turning Every Production AI Failure Into a Regression Test ## 1. The Core Idea Traditional software has a tight loop: bug reported → reproduce → write failing test → fix → test passes → merge. That failing test stays in the suite forever, so the bug can never silently come back. Most LLM/agentic systems don’t have this loop. A user hits a bad tool call, a hallucinated claim number, a broken JSON parse, or a fraud model that missed an obvious pattern — someone patches the prompt or the retrieval step, ships it, and moves on. There’s no artifact left behind that proves the failure won’t recur. Three weeks later, a prompt tweak for a different bug quietly reintroduces the first one. The **Eval Flywheel** closes this loop for AI systems: ``` Production failure → Triage & label → Minimal reproducible case →Add to eval set → Fix → Eval passes → Ship → Eval runs on every future change ``` Each failure permanently converts into a row in an eval dataset. Over time the eval set becomes a dense map of every way your system has ever broken — a regression suite grown entirely from real-world edge cases instead of hypothetical ones you imagined in advance. The “flywheel” part is the compounding effect: the more incidents you convert, the harder it becomes for old failure modes to…