{"slug": "eval-driven-development-what-it-is-how-it-differs-from-tdd-and-a-loop-that-works", "title": "Eval-driven development: what it is, how it differs from TDD, and a loop that works", "summary": "Eval-driven development (EDD) — building LLM applications and agents by writing the eval before the change and accepting a change only when it beats a baseline without regressions — traces to Vercel's October 17, 2024 post as the earliest prominent use of the phrase, according to the author's dated source review. The practice is not test-driven development, and Hamel Husain and Shankar's July 27, 2025 post answered \"Generally no\" to whether teams should practice it, while Anthropic's January 9, 2026 engineering post recommended it by name for agent teams. The author, who builds the EvalShift tool, lays out a six-step loop and warns that owning a dashboard of scores while shipping on instinct is not the same as having evals.", "body_md": "[← all posts](https://www.evalshift.dev/blog)\n\n# Eval-driven development: what it is, how it differs from TDD, and a loop that works\n\nWhat eval-driven development is, where the term comes from, how it differs from test-driven development, and a six-step loop for LLM apps and agents.\n\nEval-driven development is a way of building LLM applications and agents in which the eval comes before the change. You write down what a new capability or a fix should do as a set of tasks with checks, run them, make the change, and accept it only when the results improve against a baseline without breaking what already worked. The name is borrowed from test-driven development, and the borrowing causes most of the confusion: an eval is not a unit test.\n\nThe term is also used in two senses, and two of the best-known teachers of LLM evals advise against the stricter one. This post covers what eval-driven development means, where the term comes from, how it differs from test-driven development, a six-step loop with a worked example, the ways the practice goes wrong, and where it lives in CI. A disclosure: I build EvalShift, a tool for one part of this loop, and it has a short section near the end. The rest works with any tool or none.\n\n## What eval-driven development is\n\nEval-driven development (EDD) comes down to three habits.\n\n1. +**The expectation is written first.** Before a prompt edit, a new tool or a model swap, someone\nwrites the tasks and the checks that say what done means.\n2. +**A change is accepted on a comparison.** The new version runs on the same tasks as the old\none, and the difference decides. Reading five outputs and finding them fine does not count.\n3. +**Failures flow back.** A bug report, a bad production trace or a surprising transcript becomes\na task.\n\n\"Eval driven development\", \"eval-driven development\", \"evaluation-driven development\" and EDD all\nname the same practice. The parts of an eval (task, trial, grader, transcript, outcome) are\ndefined in [AI evals for agents](https://www.evalshift.dev/blog/ai-evals-for-agents). This post is about the order of work.\n\nHaving evals is not the same thing. A team can own a dashboard full of scores and still ship on instinct.\n\n## Where the term comes from\n\nI could not find who used the phrase first. These are the dated sources I read in November 2026, oldest first.\n\n| Date | Source | What it says | \n|---|---|---|\n| March 29, 2024 | Hamel Husain, [Your AI Product Needs Evals ↗](https://hamel.dev/blog/posts/evals/) | Does not use the term. Argues that unsuccessful LLM products almost always share one root cause, the lack of a robust evaluation system | \n| October 17, 2024 | Vercel, [Eval-driven development: Build better AI faster ↗](https://vercel.com/blog/eval-driven-development-build-better-ai-faster) | The earliest prominent use of the phrase I found. Says v0 is built this way | \n| November 21, 2024 | Xia and colleagues, [Evaluation-Driven Development of LLM Agents ↗](https://arxiv.org/abs/2411.13768) | A process model and reference architecture. The November 2025 revision renames it EDDOps and joins offline and online evaluation in one loop | \n| April 1, 2025 | Bowne-Anderson and Krawczyk, [Escaping POC Purgatory ↗](https://www.oreilly.com/radar/escaping-poc-purgatory-evaluation-driven-development-for-ai-systems/) | Teams that get past the demo stage adopt \"some version of evaluation-driven development\" | \n| July 27, 2025 | Husain and Shankar, [Should I practice eval-driven development? ↗](https://hamel.dev/blog/posts/evals-faq/should-i-practice-eval-driven-development.html) | \"Generally no\" | \n| January 9, 2026 | Anthropic, [Demystifying evals for AI agents ↗](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) | Recommends the practice by name for agent teams | \n| June 2026 | DeepEval, [Eval driven development ↗](https://deepeval.com/blog/eval-driven-development) | A three-step version, with a warning that copying test-driven development one to one leads to bottlenecks and high API costs | \n| Undated | OpenAI, [Evaluation best practices ↗](https://developers.openai.com/api/docs/guides/evaluation-best-practices) | \"Adopt eval-driven development\" is the first tip on the page | \n\n### Two meanings of the term\n\n**The loose meaning: evals decide every change.** This is how Vercel uses it. Its post describes\nevals as \"like end-to-end tests for AI and other probabilistic systems\" and says the v0 team\niterates on prompts almost daily, adds new failing prompts to the eval set, and includes eval\nresults in every pull request that touches the output pipeline. OpenAI's version is one line:\n\"Evaluate early and often.\" Its [evals guide ↗](https://developers.openai.com/api/docs/guides/evals)\ncompares the process to behavior-driven development, where behavior is specified before it is\nimplemented.\n\n**The strict meaning: the eval exists before the capability.** This is Anthropic's. Its guide\nrecommends that teams \"build evals to define planned capabilities before agents can fulfill them\"\nand then iterate until the agent performs well. Anthropic adds that it often builds features as\nbets on what models will do in a few months, and that evals starting at a low pass rate show\nwhich bets paid off when a new model arrives.\n\n### The objection\n\nHamel Husain and Shreya Shankar teach a course on AI evals, and their FAQ answers the strict meaning with \"Generally no.\" Their argument is that failure modes in ordinary software are predictable and failure modes in LLM systems are not, so you cannot anticipate what will break. Their advice: \"Write evaluators for errors you discover, not errors you imagine.\" They allow one exception, a constraint where you know exactly what success looks like, such as never mentioning a competitor.\n\nThere is research behind the objection. Shankar and colleagues\n[studied ↗](https://arxiv.org/abs/2404.12272) people building LLM-assisted evaluators in April 2024\nand named a pattern they call criteria drift: people need criteria in order to grade outputs, and\ngrading outputs is what teaches them the criteria.\n\nBoth sides are right, about different kinds of check. What an agent must have done by the end of a task, and the rules it must never break on the way, are knowable before any code exists. Whether its replies are good is not knowable until you have read a few dozen of them. The loop below writes the first kind of check before the change and the second kind after it.\n\n## Eval-driven development vs test-driven development\n\n|  | Test-driven development | Eval-driven development | \n|---|---|---|\n| Written first | A failing test for the next piece of functionality | Tasks with success criteria for the next capability or change | \n| One run returns | Pass or fail, the same every time | A pass rate. The same task can pass on one run and fail on the next | \n| \"Green\" means | Every test passes | The new capability's rate went up, and nothing else dropped by more than the noise | \n| The oracle | An assertion in code, trusted as written | Code checks, plus LLM judges that need validating against human labels | \n| Cost of a run | Milliseconds, free | Minutes and real model calls | \n| When it runs | On every save | On changes that can move the result: prompt, model, tools, the suite | \n| What changes without you | Nothing, until you bump a dependency | The model: aliases move, versions retire, the serving stack changes | \n| The third step | Refactor the code | Read the transcripts, repair the eval, graduate passing tasks | \n\n### What carries over\n\nThe discipline carries over. Writing the expectation first forces you to decide what done means.\nMartin Fowler's [description ↗](https://martinfowler.com/bliki/TestDrivenDevelopment.html) of\ntest-driven development, which Kent Beck developed in the late 1990s, includes a step people\nforget: writing out a list of test cases before the first one is implemented. That list maps\ndirectly onto a task set. So do small steps, one change between runs, and keeping every past\nsuccess as a regression test.\n\n### What does not carry over\n\n**Results are rates.** There is no red and no green. A suite of 50 cases that each pass 98% of\nthe time comes back fully green in only 36% of runs, which is why\n[LLM testing](https://www.evalshift.dev/blog/llm-testing) recommends three to five runs per case and a gate on the\ncomparison. Small sets are noisier still: 16 passes out of 20 is 80%, with a 95% interval of\nroughly 58% to 92%, so going from 16 to 17 tells you nothing on its own.\n\n**Graders need validating.** A unit test's assertion is correct by inspection. An LLM judge is a\nmodel with its own error rate, and it returns a confident verdict whether or not its criterion\nmakes sense. [When to trust an LLM judge](https://www.evalshift.dev/blog/when-to-trust-an-llm-judge) has the procedure\nfor checking one against human labels before it blocks anything.\n\n**The model changes under you.** In test-driven development the code under test changes only when\nyou change it. An LLM application depends on a model that can be re-pointed, retired or served\ndifferently while your repository stays the same, so the regression suite is also the acceptance\ntest for the provider's next release. The\n[deprecation schedule](https://www.evalshift.dev/blog/llm-model-deprecation-schedule) lists the dates you do not choose.\n\n## The eval-driven development loop\n\nSix steps. The first three happen before the change, which is the part that makes this eval-driven and not just evaluated.\n\n## Step 1: Write the change down as tasks before you build it\n\nDescribe the capability or the fix as 20 to 50 tasks, each with an input and a statement of what success means in terms of the world, not the reply. Anthropic's guide gives the same starting size and the same sources: real failures, the checks you already run by hand before a release, the bug tracker and the support queue. A capability that does not exist yet has no failures, so use real requests that people or an older system handle today.\n\nTwo properties make a task set worth running. Each task has to be unambiguous: Anthropic's test is that two domain experts would independently reach the same pass or fail verdict. And the set has to be balanced, with tasks where the behavior should happen and tasks where it should not. Anthropic learned this building web search for Claude.ai and puts it in five words: \"One-sided evals create one-sided optimization.\" If a task is hard to write, the requirement is not finished.\n\n## Step 2: Grade outcomes with code first, and leave judges for later\n\nWrite checks only for what you can know in advance:\n\n- +**The outcome.** The record exists, the status changed, the field has the right value.\n- +**The rules.** Steps that must never be skipped and actions that must never repeat.\n- +**The structure.** The output parses and validates.\n\nThese are code, they cost nothing to run, and you know what they should say before the agent exists. Do not write an LLM judge for \"helpful\" or \"on-brand\" yet. You have not seen the outputs, so the criterion would be a guess. That judge is written in step 5, from failures you have read.\n\nPick the numbers you will report now as well.\n[Agent evaluation metrics](https://www.evalshift.dev/blog/agent-evaluation-metrics) defines the candidates. For a\ncustomer-facing agent the one that matters is the share of tasks that pass on every trial.\n\n## Step 3: Run the baseline before you change anything\n\nRun the new tasks against the current system. For a new capability most of them should fail. This is the red step of test-driven development, with the same purpose: a check that cannot fail is not checking anything. Anthropic notes the opposite case too. With a frontier model, a task that fails on every one of many trials is more often a broken task than an incapable agent.\n\nThen run the existing regression suite a few times with nothing changed. The spread between those\nruns is your noise floor, and without it a two-point drop after your change cannot be read.\n[How many eval cases do you need](https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need) covers what suite size\nlets a comparison conclude.\n\n## Step 4: Make one change and compare it with the baseline\n\nChange one thing: the prompt, or a tool definition, or the model. Run the new tasks and the regression suite on the old and the new version, on the same cases, several trials each. Then read two results.\n\n- +**The capability tasks should go up.** That is the hill you are climbing.\n- +**The regression suite should not go down** by more than the noise floor from step 3.\n\nIf both the prompt and the model changed, you cannot attribute the result. Make the second change a second run.\n\n## Step 5: Read the transcripts and fix the eval too\n\nA score tells you that something moved. A transcript tells you what. Read every failure on the new tasks and a sample of the passes. Each failure is one of three things:\n\n- +**The agent is wrong.** Fix the prompt, the tool or the code, and run again.\n- +**The eval is wrong.** The grader rejected a valid solution, or the task was impossible.\nAnthropic's guide expects this: when grading is unfair or a task is ambiguous, revise the eval.\n- +**Nobody anticipated it.** This is the case Husain and Shankar describe. Add a task, and if\ncode cannot check it, write a judge now: one criterion, pass or fail, checked against 30 to 50\ntrials you labeled by hand, and advisory until that comparison holds up.[LLM-as-a-judge](https://www.evalshift.dev/blog/llm-as-a-judge) covers how.\n\nIn test-driven development the third step is refactoring the code. Here it is refactoring the eval.\n\n## Step 6: Graduate passing tasks and gate every later change\n\nWhen a capability task passes on every trial, move it into the regression suite. Anthropic calls this graduating: a task that asked whether the agent can do something now asks whether it still can. The capability set then needs refilling, because an eval at 100% tracks regressions and gives no signal for improvement.\n\nThen make the regression suite automatic: it runs on every pull request that touches the prompt,\nthe model name, the tool definitions or the suite, and compares the branch with the base branch.\nKeep feeding it, too. [LLM monitoring](https://www.evalshift.dev/blog/llm-monitoring) covers finding production failures,\nand\n[building a golden suite from production traffic](https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic)\ncovers turning recorded runs into cases.\n\n## A worked example: adding cancellation to a support agent\n\nThis example is illustrative. It shows the order of work, and none of it is a measurement.\n\nA support agent answers billing questions. The next feature lets it cancel a subscription through\na new `cancel_subscription` tool. Today those requests go to a person.\n\n**Step 1.** Pull 24 real requests from the support queue and write each as a task. Twelve are\nplain requests to cancel. Eight are requests where the agent must not cancel: a question about\nwhat cancelling would do, a request to pause, an account the customer does not own. Four are\nphrased in ways the prompt will not expect.\n\n```\n- id: cancel-explicit-monthly\n  input: \"Please cancel my subscription. Account 7741.\"\n  expect: {account: \"7741\", status: cancelled, cancel_calls: 1}\n\n- id: cancel-question-only\n  input: \"If I cancelled today, would I lose my saved reports?\"\n  expect: {account: \"7741\", status: active, cancel_calls: 0}\n```\n\n**Step 2.** Three code checks, one result each: the outcome in the sandbox database, the number\nof cancel calls, and the rule that a lookup comes before a cancel. No judge yet, because nobody\nhas seen what the agent says when it cancels.\n\n``` python\ndef grade(trial, db, expect):\n    names = [call.name for call in trial.tool_calls]\n    cancels = names.count(\"cancel_subscription\")\n    first_cancel = names.index(\"cancel_subscription\") if cancels else len(names)\n\n    return {\n        \"outcome\": db.subscription(expect[\"account\"]).status == expect[\"status\"],\n        \"cancel_count\": cancels == expect[\"cancel_calls\"],\n        \"lookup_first\": cancels == 0 or \"lookup_subscription\" in names[:first_cancel],\n    }\n```\n\n**Step 3.** Run the 24 tasks on the current agent, three trials each. The tool does not exist, so\nthe twelve should-cancel tasks fail on `outcome` by construction, and the eight should-not tasks\nshould pass. If one of the eight fails, the task or the grader is wrong, and you found out before\nwriting any feature code.\n\n**Step 4.** Add the tool and one paragraph to the system prompt, and run both sets on the old and\nthe new version. Suppose the should-cancel tasks now mostly pass and two of the should-not tasks\nfail, because the agent cancelled when a customer asked a hypothetical question. Twelve\nhappy-path tasks alone would have reported a clean success.\n\n**Step 5.** Read the transcripts. Suppose they show two things nobody wrote a task for: a\ncustomer with two subscriptions gets the wrong one cancelled, and the replies never say when\naccess ends. The first becomes three new tasks with an outcome check. The second cannot be\nchecked by code, so it becomes a judge with one criterion, labeled against 30 trials and kept\nadvisory.\n\n**Step 6.** Tasks that pass on all three trials move to the regression suite, and the ones that\nstill fail stay in the capability set. The cancellation rules now run on every pull request,\nincluding the one that swaps the model next quarter.\n\n## How eval-driven development goes wrong\n\n### Tuning the prompt to the eval set\n\nTwenty rounds of prompt edits against the same 30 tasks produce a prompt that passes those 30\ntasks. The authors of\n[What We've Learned From a Year of Building with LLMs ↗](https://applied-llms.org/) open their\nwarning about over-weighted evals with Goodhart's law: \"When a measure becomes a target, it\nceases to be a good measure.\" Three habits limit the damage. Keep a held-out slice of tasks that\nyou run and never read failures from. Refill the set from production every few weeks. And treat\na prompt that names specific test inputs as a bug.\n\n### A score nobody reads\n\nA pass rate on a dashboard is not a decision. If no change has ever been blocked or reverted because of the suite, the suite is decoration. The fix is steps 5 and 6: someone reads the transcripts, and the comparison sits on the pull request where the change is reviewed.\n\n### A judge nobody validated\n\nOpenAI lists not calibrating automated metrics against human evals among its anti-patterns. An unvalidated judge fails in both directions: it blocks good changes, which teaches the team to override it, and it passes bad ones with a reassuring number attached.\n\n### An absolute threshold that breaks when the suite grows\n\n\"Pass rate must stay above 90%\" works until the suite improves. Take 100 cases passing at 92%. Add 15 hard cases from production, of which the agent passes 5. The suite is now better and the pass rate is 84.3%, with no change to the agent. The gate turns red, and the team learns to stop adding hard cases. Gate on the comparison with the base branch, and keep thresholds for budgets that are meant to be absolute, such as zero violations of a rule.\n\n### Evals written after the fact\n\nAn eval written after the feature ships encodes what the system does, not what it should do. Anthropic's version: \"Evals get harder to build the longer you wait.\" It is still worth doing: the same guide says the Bolt team started after its agent was widely used and had an eval system in three months. Start from failures, not from current behavior.\n\n## Where the loop lives in CI\n\nThe loop runs at three speeds.\n\n| Speed | What runs | When | What it decides | \n|---|---|---|---|\n| Inner loop | The capability tasks for the change in hand | By hand, many times a day | Whether to keep iterating | \n| Pull request | The regression suite, branch against base | On changes to the prompt, model, tools or suite | Whether the change may merge | \n| Schedule | The regression suite on the pinned production model | Daily or weekly | Whether anything moved with no change of yours | \n\nVercel's post describes the middle row: every pull request that affects the output pipeline\ncarries eval results. OpenAI's guide calls it continuous evaluation. Restrict the trigger to the\nfiles that can move the result, because every run costs model calls.\n[LLM regression testing in CI](https://www.evalshift.dev/blog/llm-regression-testing-in-ci) walks through a gate on GitHub\nActions, and [prompt regression testing](https://www.evalshift.dev/blog/prompt-regression-testing) covers where the prompt\ntext has to live for a prompt edit to stay comparable with the base branch.\n\nKeep the tasks and the graders in the repository, next to the prompt they test. The OpenAI page that recommends eval-driven development carries a notice, as of November 2026, that OpenAI is deprecating its hosted Evals platform: read-only for existing users from October 31, 2026, and scheduled to shut down on November 30, 2026. A suite that lives as files survives a change of vendor.\n\n## Eval-driven development with EvalShift\n\nDisclosure again: I'm the founder. [EvalShift](https://www.evalshift.dev/docs) covers the regression half of the loop:\nsteps 3, 4 and 6 for a model change, and the pull-request gate. It is not a harness for\ncapability evals on a feature that does not exist yet. The CLI runs a fixed suite on two models,\nthe current one and a candidate, and it never runs your tools: it replays recorded runs, feeding\nboth models the tool results captured in production. An outcome check against a database, like\nthe one in the worked example, stays in your own tests.\n\nThe capture SDK records real runs, and `evalshift capture sync --rounds all` turns them into a\n[golden suite](https://www.evalshift.dev/docs/golden-suite) that carries each run's recorded tool results. The rules from\nstep 2 go into the config as trace rules that both models are held to:\n\n```\ndefaults:\n  samples_per_example: 3   # every case runs three times on each model\n\nevaluators:\n  trace_invariants:\n    - name: cancellation_contract\n      owner: \"@support-team\"\n      rules:\n        - id: lookup-before-cancel\n          type: order\n          before: lookup_subscription\n          after: cancel_subscription\n        - id: one-cancel\n          type: call_count\n          tool: cancel_subscription\n          max_calls: 1\n\nmigration_policy:\n  max_invariant_violations: 0\n```\n\n`evalshift compare --policy-gate` runs both models over the suite, scores each side, tests whether\nthe differences are larger than the noise, and exits non-zero when the\n[migration policy](https://www.evalshift.dev/docs/migration-policy) returns fail or conditional pass. Judges that\n`evalshift init` writes start as advisory (`blocking: false`), which matches step 5. The\n[GitHub Action](https://www.evalshift.dev/docs/action) runs the suite on a pull request, pushes the run, compares it with a\nbaseline run on the base branch, and by default gates on the policy verdict. The CLI and SDK are\nopen source and run locally with your own provider keys.\n\n## FAQ\n\n**What is eval-driven development?** Eval-driven development is a way of building LLM\napplications and agents in which the eval is written before the change. You describe a new\ncapability or fix as tasks with checks, run them against the current system, make the change, and\naccept it only when results improve against that baseline without regressing what already worked.\n\n**What is the difference between eval-driven development and test-driven development?** Both\nwrite the expectation before the code. In test-driven development a test passes or fails the same\nway every time and the assertion is trusted. In eval-driven development a result is a pass rate\nthat varies between runs, LLM judges need validating against human labels, each run costs model\ncalls, and the model can change while your code stays the same.\n\n**Where does the term eval-driven development come from?** Its origin is unclear. One of the\nearliest prominent uses is a Vercel post from October 17, 2024 describing how v0 is built. A paper\non evaluation-driven development of LLM agents followed on November 21, 2024, OpenAI's evaluation\nbest practices recommend it, and Anthropic recommended it for agent teams on January 9, 2026.\n\n**Should you write evals before building a feature?** Write the checks you can know in advance:\nthe outcome a task must reach and the rules that must never be broken. Hamel Husain and Shreya\nShankar advise against writing evaluators for imagined errors, so leave quality criteria and LLM\njudges until you have read real outputs and seen what fails.\n\n**How many evals do you need to start eval-driven development?** 20 to 50 tasks drawn from real\nfailures or real requests are enough to start. Balance them between cases where a behavior\nshould happen and cases where it should not. A set that small is noisy: 16 passes out of 20 has a\n95% interval of roughly 58% to 92%.\n\n**How do you avoid overfitting a prompt to your evals?** Keep a held-out slice of tasks that you\nrun but never read failures from, refill the suite from production traffic every few weeks, and\ntreat a prompt that names specific test inputs as a bug. Gate on a comparison with the base\nbranch, not on an absolute score.\n\n## Keep reading\n\n- +[AI evals for agents](https://www.evalshift.dev/blog/ai-evals-for-agents) : the parts of an eval and seven steps to write\none.\n- +[LLM testing](https://www.evalshift.dev/blog/llm-testing) : the types of LLM tests and why every case is a pass rate.\n- +[LLM regression testing in CI](https://www.evalshift.dev/blog/llm-regression-testing-in-ci) : step 6 as a pull-request\ncheck.\n- +[Prompt edits deserve the same gate as a model swap](https://www.evalshift.dev/blog/prompt-regression-testing) : keeping\nprompt changes comparable with the base branch.\n- +[How many eval cases do you need?](https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need) : sizing the suite so a\ncomparison returns a verdict.", "url": "https://wpnews.pro/news/eval-driven-development-what-it-is-how-it-differs-from-tdd-and-a-loop-that-works", "canonical_source": "https://www.evalshift.dev/blog/eval-driven-development", "published_at": "2026-11-13 21:57:24.839046+00:00", "updated_at": "2026-11-13 21:57:27.578767+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "ai-agents", "ai-tools", "mlops"], "entities": ["EvalShift", "Hamel Husain", "Shankar", "Vercel", "Anthropic", "DeepEval", "OpenAI", "Xia"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/eval-driven-development-what-it-is-how-it-differs-from-tdd-and-a-loop-that-works", "markdown": "https://wpnews.pro/news/eval-driven-development-what-it-is-how-it-differs-from-tdd-and-a-loop-that-works.md", "text": "https://wpnews.pro/news/eval-driven-development-what-it-is-how-it-differs-from-tdd-and-a-loop-that-works.txt", "jsonld": "https://wpnews.pro/news/eval-driven-development-what-it-is-how-it-differs-from-tdd-and-a-loop-that-works.jsonld"}}