Member-only story
1. The Core Idea #
Traditional software has a tight loop: bug reported → reproduce → write failing test → fix → test passes → merge. That failing test stays in the suite forever, so the bug can never silently come back.
Most LLM/agentic systems don’t have this loop. A user hits a bad tool call, a hallucinated claim number, a broken JSON parse, or a fraud model that missed an obvious pattern — someone patches the prompt or the retrieval step, ships it, and moves on. There’s no artifact left behind that proves the failure won’t recur. Three weeks later, a prompt tweak for a different bug quietly reintroduces the first one.
The Eval Flywheel closes this loop for AI systems:
Production failure → Triage & label → Minimal reproducible case →Add to eval set → Fix → Eval passes → Ship → Eval runs on every future change
Each failure permanently converts into a row in an eval dataset. Over time the eval set becomes a dense map of every way your system has ever broken — a regression suite grown entirely from real-world edge cases instead of hypothetical ones you imagined in advance. The “flywheel” part is the compounding effect: the more incidents you convert, the harder it becomes for old failure modes to…