Agents are now users of software, doing work on people's behalf. Whether you're building the agent or the software it uses (API, CLI, MCP server, SDK), evals give you a way to measure how well that work gets done.
Agents probably missed your launch video and billboard on Highway 101, but they still go through the same funnel a human does to find and use a product: Discovery, Selection, Access, and Success.
You can build evals for any stage of this funnel. Here, we’ll focus on the final stage, Success: whether the agent completes the user’s task correctly.
That's more difficult to check than it sounds. An agent can place a “successful” order for a hot latte when the user asked for an iced one. The request worked, so the logs in your Observability stack look fine. And because LLM outputs vary between runs, one good manual test can’t tell you whether a fix worked. You need to repeat the same tasks and grade the outcomes, before and after the change.
That’s what an eval can do for you. Here’s how we build one.
If a test asks “did it work?” an eval asks “how often does it work, and how well?” You give an agent a task, grade the outcome, and aggregate the results across repeated attempts. We group agentic software tests by how many LLM calls they typically need: deterministic tests (0), one-turn evals (1), and E2E evals (2+ across a whole task). This has turned out to be a great proxy for the effort to build them, the time they take to run, and the cost of running them. Just like in the practical test pyramid, the cheap and fast tests should make up the bulk of your test suite.
To use the pyramid, start with the problem you want to catch and pick the lowest tier that can catch it. For Success, that's the top. A one-turn eval checks a single decision, like which command the agent runs first. But Success depends on the whole sequence of steps (e.g. search for cafes → filter on user criteria → place the latte order). Each one can look reasonable on its own, and the agent can still get the wrong outcome.
An E2E eval lets the agent work through the whole task in an environment, then checks whether the final result meets the user’s requirements. You’ve probably seen a few already; most popular coding benchmarks, like SWE-bench and Terminal-Bench, are E2E evals.
Say you own SF Beans, an API for ordering coffee in San Francisco. One of the best ways to give agents access to your service is through a CLI, so you’re shipping sfbeans.
Here’s a typical request from one of your customers:
Get me a large iced latte with whole milk, ready for pickup by 9:00, under $8 including tip. I’m in the Mission.
Here’s what sfbeans knows at 8:30am:
Fog City Roasters: ice machine is down, so no iced drinks today (temperature)
24th St Beans: latte is $7.50 on the menu, but $9.27 after tax and tip (budget)
**Dolores Drip:** latte is $5.50, but the earliest pickup is 9:20 *(time)*
**Valencia Grind:** $7.42 after tax and tip, ready at 8:50 *(meets all three)*
Here’s what you could test at each layer of the pyramid for sfbeans:
We’ll build the coffee eval with Harbor, an open-source framework from the creators of Terminal-Bench that runs agents in containers, checks results, and records their actions.
Here are the terms we’ll use:
Task: one problem for the agent to solve, with a starting environment and a way to check whether it succeeded
Variant: the agent setup you’re testing, like the model, prompt, tools, memory and skills
Trial: one attempt by one variant at one task
Run (a job in Harbor): every variant tried on every task, several times each
Trace (a trajectory in Harbor): everything recorded during a trial: messages, tool calls, logs, files and final output
Building E2E evals for everything isn’t practical. With traditional software, you can aim for 100% test coverage. With agents, the surface you’d need to cover is everything an LLM might do with your product, and even figuring out where that ends is hard (see the jagged frontier).
Start with one outcome your users care about, narrow enough to evaluate with a handful of tasks. You’ll usually find that goal in one of three places:
Observability: which agents use your product, and where they fail
Customers: support tickets, feedback, and churn
Roadmap: the product hypotheses you most need to validate
For example, your goal might be to: Increase the success rate on a specific task
Reduce the number of steps required to accomplish a goal
Check whether an agent skill helps, and get feedback to improve it A/B test a prompt
Explore the frontier by checking whether a use case is achievable with current LLMs, and if they fail, why
We looked in all three places for sfbeans:
Observability: most agent traffic comes from Claude Code and Codex. We know because sfbeans adds a user-agent header when it detects a specific coding harness.
Customers: the top support ticket is “my agent ordered the wrong drink”: a hot latte instead of iced, a pickup after the user’s meeting, or a total over budget once tax and tip are added.
Roadmap: nothing urgent, so the wrong-drink problem goes first, since it drives churn.
That gives us our goal:
| Goal | Increase the success rate on ordering the exact drink the user asked for |
|---|---|
| Agents | Claude Code and Codex, since that’s where our traffic comes from. |
| In scope | Drink, pickup time, and budget (incl. tax and tip), using today’s sfbeans . |
| Out of scope | Cancellations, refunds, and accounts. We’ll eval those later. |
| Done when | Both agents get the order right in every trial, not just most of the time. |
Here are the metrics we use most for E2E agent evals:
Success
Success rate: the share of trials that pass
Pass@k: the chance that at least one of k trials passes. Answers “can the agent do this at all?”
Pass^k: the chance that all k trials pass. Answers “can I rely on it every time?” Efficiency (measured on successful trials only)
Steps: the number of agent turns needed to complete the task
Time: how long the trial took
Cost: how much the trial cost, in dollars
Our goal is to get every order right in every trial. Once we build and run the eval in Steps 3 and 4, here’s what five trials of Claude Code + Opus 5.5 (anthropic/claude-opus-5.5 on AI Gateway) on the iced latte order could look like:
| Trial | Result | What happened | Steps | Time | Cost |
|---|---|---|---|---|---|
| 1 | Pass | Compared quotes from 3 cafes, ordered at Valencia | 9 | 3:40 | $0.52 |
| 2 | Fail: over budget | Skipped quote, ordered at 24th St: $9.27 | 5 | 1:50 | $0.31 |
| 3 | Pass | Compared quotes from 4 cafes, ordered at Valencia | 11 | 4:30 | $0.68 |
| 4 | Fail: late | Ordered at Dolores, ready 9:20 | 6 | 2:10 | $0.35 |
| 5 | Pass | Skipped Fog City, ordered at Valencia | 8 | 3:20 | $0.49 |
What the numbers mean:
Success rate: 3 of 5 trials passed, so 60%.
Pass@5: assuming independent trials with a 60% success rate, 1 − 0.4^5 ≈ 99%. Give the agent 5 tries and it almost certainly gets it right once.
Pass^5: 0.6^5 ≈ 8%. The chance the agent gets 5 orders in a row right.
Steps: (9 + 11 + 8) / 3 ≈ 9 on successful trials. The failed trials are shorter because the agent skipped work, like quoting, which is why efficiency only counts successful trials.
Time: ~3:50 on successful trials. Cost: $0.56 per successful trial, but $2.35 / 3 = $0.78 per success once you count the failed trials you paid for too.
So does Opus work? Its 99% pass@5 says yes. Its 8% pass^5 says a user who orders this every weekday gets at least one wrong coffee in 92% of weeks. Both numbers are right. They just answer different questions: pass@5 asks whether the agent can get the order right, and pass^5 asks whether it gets it right every time.
Which metric matters most depends on what a failure costs your user. Someone ordering their morning coffee doesn’t get a retry. If the agent orders a hot latte, or one that’s ready after their train leaves, they’ve paid for the wrong coffee and started their day annoyed. They need it right every morning, so sfbeans optimizes for pass^5.
Other products are more forgiving, because a failed attempt is cheap and gets caught before it matters:
| Product | What a failed attempt costs the user | Metric to lead with |
|---|---|---|
| Order my morning coffee | Paid for the wrong drink, late for work, lost trust | Pass^k |
| Coding agent with a test suite | Extra time and model cost; the agent uses test feedback to revise its code and try again | Pass@k |
| Draft an email the user reviews | A quick edit before hitting send | Success rate |
Overall averages can hide a task that fails frequently, so always inspect results per task. And with only five trials, 100% means “no failures seen yet,” not “never fails.”
In Step 1, we set a goal for the eval: increase how often agents order the exact drink the user asked for. But an agent can’t attempt a goal. It needs a task: one concrete request, drawn from what your users actually ask for.
Pick tasks hard enough that you learn something from running them. A good set covers the common case, the edge cases that trip agents up, and at least one request where the right move is not to act.
For each task, define these parts: An instruction: the user’s request
An environment: where the agent works, with the same starting state for every trial
A verifier: a check of the final state, ideally scored per constraint
A known-good solution (optional): a reference answer that should pass the verifier. Skip it for tasks with no valid answer, like impossible-request
You can make every aspect of that environment as fake or as realistic as you need, depending on your goals. The OpenPipe team has a nice writeup on how they defined the environment for an email research agent.
We wrote four tasks, each one a Harbor task folder in tasks/.
Common case: iced-latte-mission, the order from earlier
Edge case: usual-unavailable, where the user’s usual cafe is out of their drink
Edge case: team-order, with 6 drinks, $45 total, and one pickup spot
Don’t act: impossible-request, where nothing fits, so the right move is to tell the user
impossible-request has one extra check. Its verifier reads a command log the agent can’t write to, to confirm the agent searched before giving up. We learned we needed that the hard way (see Step 5).
Here’s iced-latte-mission:
The instruction gives the agent the user’s request and names the sfbeans CLI. In production, agents find the tool through a skill, docs, or an MCP server. Naming it here skips Discovery and Selection, keeping the eval focused on Success.
We also specify that it’s 8:30am. Only sfbeans uses our frozen clock; the sandbox’s system clock still shows the real time:
The environment is part real, part fake. The agent uses the real sfbeans CLI, but against seeded cafes and a frozen clock, so every trial starts on the same morning:
To keep the demo simple, cafes.json lives inside the agent’s environment. In a real eval, protect the seed data and verifier code, and make sure the agent can update order records only through the CLI. Agents can attempt to reward hack if it helps them pass, for example by editing the inventory or writing orders.json directly.
For this coffee order, the verifier checks the final order against the user’s requirements. Harbor runs test.sh after the agent finishes and reads the scores it writes to /logs/verifier/reward.json:
The actual check lives in check_orders.py. It scores each constraint separately, then sets reward to 1 only if all of them pass. Harbor uses reward as the trial’s main score:
The solution is a known-good answer. Run it before any real agent to prove the task works:
Finally, task.toml ties the task together. It sets the timeouts, runs the verifier in the agent’s sandbox (where orders.json lives), and saves the orders as an artifact so we can inspect them in Step 5:
If you share the universal distaste for slow CI, you’ll hate waiting for evals. Every trial waits on an agent making LLM calls, step after step, and a run repeats that for every variant and task, several times each. Harbor’s Vercel integration, which we built upstream, speeds that up a lot. It runs trials concurrently, each in its own Vercel Sandbox, and caches built task environments so they aren’t rebuilt for every trial. Routing model calls through AI Gateway gives you one key and one budget across model providers.
Two agents, Claude Code and Codex, with two models each, make four variants.
Then install Harbor, connect a Vercel project, and run the job:
This run fits on the Hobby plan: 10 concurrent sandboxes is the Hobby limit, and each trial is well under its 45-minute session cap. On Pro, you can raise n_concurrent_trials as high as 10,000 (Sandbox limits).
To keep the example simple, we pass the OIDC token into each sandbox as an env var, where the agent can read it. Beyond a demo, use short-lived tokens, and use Harbor’s Vercel integration to set Sandbox firewall rules that lock each trial down to only the hosts it needs, like AI Gateway.
Internally, we go a step further and use credentials brokering to attach credentials to outgoing requests, keeping API keys outside the agent’s environment.
Now for the meaty part: trace-diving. Scores tell you how often your agents fail; traces tell you why. Read the failures, and some passes too, since a pass only means the verifier was satisfied.
The run from Step 4 leaves 80 trial folders in jobs/sfbeans-baseline/. Each has the trial’s result, the agent’s trajectory, the verifier’s scores, and, because task.toml saves it as an artifact, the order the agent placed:
Run harbor view jobs to browse them. Here’s what our four variants could report (numbers are illustrative):
| Variant | Success | Pass^5 | Avg steps (successful) | Cost per success (incl. failed trials) |
|---|---|---|---|---|
| Claude Code + Opus 5.5 | 85% | 60% | 9 | $0.61 |
| Claude Code + Sonnet 5.5 | 75% | 52% | 11 | $0.27 |
| Codex + GPT-6 Astra | 80% | 37% | 8 | $0.48 |
| Codex + GPT-6.1 Sol | 65% | 11% | 13 | $0.22 |
That’s our baseline. On pass^5, the metric we chose in Step 2, even the best variant is far from reliable. But the score doesn’t tell us what to fix. For that, we have to open the failed trials.
Take trial 2 from Step 2. Claude Code + Opus 5.5 ordered at 24th St Beans for $9.27. The verifier tells us which constraint it broke:
Right drink, on time, over budget, and the saved orders.json confirms the $9.27 total. Now we read agent/trajectory.json. The agent searches for lattes in the Mission, sees $7.50 on the menu, and goes straight to order. No quote. It treated the menu price as the final price.
Trial 1, from the same variant, quotes three cafes, rules out 24th St Beans, and orders at Valencia Grind for $7.42. So the CLI can get the order right, but the agent doesn’t reliably check the total before spending the user’s money.
We then look for that pattern across other failed trials, and if it keeps showing up, we have something specific to fix: make the total harder to miss, or add a --max-total flag so order refuses anything over budget. That’s a much more useful conclusion than “Opus scored 85%.”
When we first ran this demo, the traces also exposed problems with what the eval was measuring:
Our first run scored 0% on every variant. The agents didn’t know sfbeans existed, so they never tried it (obviously). That’s the funnel’s Discovery stage failing in our own eval. Naming the tool in the instruction fixed it, and it’s a good reminder that being discoverable is part of your product. In the wild, that’s the job of your skill, docs, MCP server and AEO.
Codex kept looking for sfbeans as a connected tool and never tried running it in the shell. That’s an Access problem, not a Success one.
Codex passed impossible-request almost every time, but when we read the traces, 9 out of 10 passes were the agent giving up without looking. The verifier only checked that no order was placed, so we tightened it to also check that the agent searched first. Without reading the traces, we’d have counted those as wins.
For larger runs, inspecting every trial by hand gets slow, so we automate it.
Up to 40 reader agents running fast models like deepseek/deepseek-v4.1-flash read trial transcripts in parallel and summarize their findings. A smarter coordinator agent reviews those summaries, sends readers back for more context when needed, and writes the final report. An analysis.md file tells the coordinator what to look for, so we can reuse the workflow across runs.
Model calls go through AI Gateway. Trial artifacts stay in Vercel Drives during analysis, and traces go to Vercel Blob for long-term storage.
Your first eval run gives you a baseline. Now act on it.
Pick the fix the traces point to, like the --max-total flag, rerun the same job, and compare against the baseline. Check whether it reduces the targeted failures without introducing new ones, then use those results to decide whether the change is ready to ship.
Once you’ve built an eval, it keeps paying off for a lot of different things. Rerun it on every release to catch regressions; test prompt, skill, and model changes before shipping; learn how agents use your product; or build auto-optimization loops on top of it.
Expect to add harder tasks over time. As models and your product improve, evals saturate, meaning agents start passing every task. That’s great for regression detection, but it can't tell you what to improve next.
Evals are the infrastructure you use to make your product better, again and again. Vercel has the pieces you need to build them in one place: AI Gateway for model calls across providers, Sandbox to run trials at scale, and Drives and Blob to keep every trace.
Ready to run your first eval? Paste this prompt into your coding agent, and it’ll build and run one for your product on Vercel with you: