{"slug": "measuring-agent-reliability-pass-k-pass-k-and-llm-judges", "title": "Measuring Agent Reliability: pass@k, pass^k, and LLM Judges", "summary": "An agent that passes 75 out of 100 test runs has a 75% pass@1 rate, but reliability depends on the product: pass@k (at least one success in k tries) yields 98.4% for k=3, while pass^k (all k succeed) yields only 42.2%, according to the τ-bench paper. The article, part two of a three-part series on evaluating LLM applications, advises choosing metrics based on user experience and warns against quoting pass@k without product context.", "body_md": "# Measuring Agent Reliability: pass@k, pass^k, and LLM Judges\n\nAn agent passes 75 out of 100 test runs. Is it reliable?\n\nThe answer depends on the product. A coding tool that can generate ten solutions and automatically verify them may only need one good result. A customer support agent does not get ten invisible attempts with every customer. It needs to work consistently.\n\nThat difference is easy to lose inside one average pass rate.\n\nThis is part two of a three-part series on evaluating LLM applications. [Part one introduces the eval pyramid](/blog/eval-pyramid): deterministic checks at the base, repeated task runs in the middle, then model and human judgment near the top. This post explains how to measure those middle and upper layers. [Part three implements every layer in a runnable starter](/blog/build-the-eval-pyramid).\n\n## TLDR\n\n- Run important tasks several times. One successful run proves capability, not reliability.\n- Use\n`pass@1`\n\nfor the normal first-try experience. - Use\n`pass@k`\n\nwhen the real product can try several times and keep any successful result. - Use\n`pass^k`\n\n(read as \"pass to the power k\") when every run needs to succeed. - Use LLM judges for qualities that code cannot measure, then test those judges against human decisions.\n\n## One run is an anecdote\n\nSuppose an agent passes 75 out of 100 independent test runs. Its observed first-try success rate is 75%. That is `pass@1`\n\n.\n\nNow ask a product question. Can the system try three times and use any successful result? Or will three customers each expect it to work on their first attempt?\n\nThose questions lead to opposite metrics.\n\n** pass@k asks whether at least one of k attempts succeeds.** It is useful when the product can generate several candidates, retry safely, or verify the good result. Code generation popularized this metric through\n\n[HumanEval](https://arxiv.org/abs/2107.03374). If a coding model produces ten solutions and one passes the tests,\n\n`pass@10`\n\nrecords success for that problem.** pass^k asks whether all k attempts succeed.** The\n\n[introduced this metric to show the reliability of agents across repeated interactions. A support agent that succeeds once out of ten is not reliable, even if its](https://arxiv.org/abs/2406.12045)\n\n`τ`\n\n-bench paper`pass@10`\n\nlooks excellent.For a task with a stable 75% success probability, assuming each run is independent:\n\n```\npass@3=1 − (1 − 0.75)3 = 98.4%\npass3=0.753 = 42.2%\n```\n\nSame agent. Same three runs. `pass@3`\n\nsays you will almost certainly find one success if you can keep trying. `pass^3`\n\nsays fewer than half of three-run groups will succeed every time.\n\nThis is why a report should never quote `pass@k`\n\nwithout describing the product. If users only get one attempt, `pass@10`\n\nis a capability demo, not their experience. If the system really can produce ten candidates and verify them, `pass@10`\n\nmay be the right operational metric.\n\n## Choose the metric from the user experience\n\nStart with `pass@1`\n\n. It is the easiest number to explain and usually the closest to what one user sees.\n\nAdd `pass@k`\n\nonly when retries or candidate selection exist in the real workflow. Do not add invisible retries to the evaluation just because they improve the score. A retry also has a cost, adds latency, and can repeat an unsafe action.\n\nAdd `pass^k`\n\nwhen consistency matters across equivalent runs. It is especially useful for customer-facing agents, long workflows, and actions where a rare failure is expensive. Choose `k`\n\nbased on the product. Ten repeated runs might be enough to expose instability during development, while a higher-risk release gate may need more evidence.\n\nKeep the runs independent when possible. Shared caches, leftover files, rate limits, or a database changed by an earlier run can make the results depend on one another. [Anthropic recommends clean, isolated environments](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) so each trial measures the agent instead of test contamination.\n\n## The implementation math, if you need it\n\nThe 75% example above assumes one stable probability. Real eval suites contain easy and hard tasks, so calculate the metric for each task and then average across tasks.\n\nRun each task `n`\n\ntimes. If `c`\n\nof those runs pass, the finite-sample estimators used by [ τ-bench](https://arxiv.org/abs/2406.12045) are:\n\n```\npass@k=1 − C(n − c, k) / C(n, k)\npassk=C(c, k) / C(n, k)\n```\n\nCalculate each expression per task, then average the task-level values.\n\n`choose(a, b)`\n\nmeans the number of ways to select `b`\n\nitems from `a`\n\n. The first formula counts groups with at least one pass. The second counts groups containing only passes.\n\nAlso report how much evidence sits behind the percentage. A 90% pass rate from 10 runs means nine successes. A 90% pass rate from 1,000 runs is much stronger evidence. Confidence intervals or bootstrap intervals make that uncertainty visible.\n\n## When a pass cannot be reduced to code\n\nSome requirements need judgment. Was the answer grounded in the source? Did it explain the caveat clearly? Was the response useful without being needlessly long?\n\nAn LLM judge can read the task, answer, source material, and a focused rubric, then return a score. This is useful when several different answers can all be correct. It is also another model that can fail.\n\nThe [MT-Bench paper](https://arxiv.org/abs/2306.05685) found that strong LLM judges could match human preference judgments well in its tested setting. It also found position and verbosity biases. A judge may prefer the first answer it sees or reward a longer response because it looks more complete.\n\nTreat a judge like production code:\n\n- Give it one clear criterion at a time.\n- Define each score using observable evidence.\n- Test it on strong, weak, and borderline examples labeled by domain experts.\n- Swap answer order in pairwise comparisons and flag inconsistent decisions.\n- Version the judge model, prompt, rubric, and examples.\n- Keep people involved for high-risk cases and disagreements.\n\n[OpenAI's grader guidance](https://developers.openai.com/api/docs/guides/graders) recommends comparing a model grader with human-labeled examples and checking for grader hacking. If your system learns one judge's quirks, the score can rise while the product gets worse.\n\n## A practical build order\n\n### 1. Define the job in product language\n\nWrite the task, expected result, forbidden outcomes, and acceptable variation. \"Resolve an eligible refund without violating policy\" is clearer than \"score above 0.8 on helpfulness.\"\n\n### 2. Start with a small set of real failures\n\nUse support tickets, bug reports, manual test cases, and failed production traces. [Anthropic suggests starting with 20 to 50 tasks](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). [OpenAI describes reviewing 50 to 100 outputs](https://openai.com/index/evals-drive-next-chapter-of-ai/) to build an initial error taxonomy. These are starting points, not universal thresholds.\n\n### 3. Use the cheapest faithful check\n\nUse schemas for structure, code for calculations, unit tests for executable behavior, and state comparisons for completed actions. Add a model judge only when semantic judgment is part of the requirement. One task can use several checks.\n\n### 4. Repeat the task and review the failures\n\nRecord the model version, prompt, tool definitions, retrieved context, cost, and sequence of actions. Group failures by cause: wrong tool, bad arguments, retrieval miss, policy violation, unsupported claim, or test-environment failure. The categories tell you what to fix.\n\n### 5. Build a layered release gate\n\nA support agent might require every schema and permission check to pass, zero forbidden actions, no decline in `pass@1`\n\n, an acceptable `pass^3`\n\nfor important workflows, and calibrated judge scores for explanation quality. A high-risk case still goes to a person.\n\nThe threshold depends on impact and reversibility. A restaurant recommendation and a bank transfer should not share a reliability bar.\n\nKeep sampling real traffic after release. Offline evals protect the failures you already know. Production shows you what the test set missed. [OpenAI calls this loop Specify, Measure, Improve](https://openai.com/index/evals-drive-next-chapter-of-ai/).\n\nCapability asks whether the system can succeed. Reliability asks whether users can depend on it. `pass@k`\n\nand `pass^k`\n\nseparate those questions, while calibrated judges cover the parts of quality that code cannot express. Pick each metric from the way the product actually behaves, not from the number that looks best in a launch review.\n\nNext: [build the complete eval pyramid](/blog/build-the-eval-pyramid) with the open-source [Eval Pyramid Starter](https://github.com/bharadwaj-pendyala/eval-pyramid-starter). The repository includes the finite-sample estimators above, isolated repeated trials, deterministic graders, judge calibration, release gates, reports, and a human review queue.\n\n## Glossary\n\n**Trial.** One attempt by the system on one eval task.The observed chance of success on one attempt.`pass@1`\n\n.The chance that at least one of`pass@k`\n\n.`k`\n\nattempts succeeds.The chance that all`pass^k`\n\n.`k`\n\nattempts succeed.**LLM judge.** A model prompted with a rubric to grade another system's output.**Golden set.** A curated set of examples and expert labels used to test a grader or system.**Error taxonomy.** A set of recurring failure categories found by reviewing failed runs.\n\n## References and further reading\n\n[Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). Repeated trials, clean environments, task design, and agent grading.[OpenAI: How evals drive the next chapter in AI for businesses](https://openai.com/index/evals-drive-next-chapter-of-ai/). Contextual product evals and the Specify, Measure, Improve loop.[OpenAI API: Graders](https://developers.openai.com/api/docs/guides/graders). Grader types, calibration, and grader hacking.[Evaluating Large Language Models Trained on Code](https://arxiv.org/abs/2107.03374). The HumanEval paper and the`pass@k`\n\nestimator.. Final-state evaluation and the`τ`\n\n-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains`pass^k`\n\nreliability metric.[Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685). Model judges, human agreement, and known biases.", "url": "https://wpnews.pro/news/measuring-agent-reliability-pass-k-pass-k-and-llm-judges", "canonical_source": "https://bharad.dev/blog/measuring-agent-reliability", "published_at": "2026-08-09 00:00:00+00:00", "updated_at": "2026-08-10 07:20:10.473454+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-agents"], "entities": ["Anthropic", "HumanEval", "τ-bench"], "alternates": {"html": "https://wpnews.pro/news/measuring-agent-reliability-pass-k-pass-k-and-llm-judges", "markdown": "https://wpnews.pro/news/measuring-agent-reliability-pass-k-pass-k-and-llm-judges.md", "text": "https://wpnews.pro/news/measuring-agent-reliability-pass-k-pass-k-and-llm-judges.txt", "jsonld": "https://wpnews.pro/news/measuring-agent-reliability-pass-k-pass-k-and-llm-judges.jsonld"}}