{"slug": "build-the-eval-pyramid-a-starter-harness-for-llm-and-agent-testing", "title": "Build the Eval Pyramid: A Starter Harness for LLM and Agent Testing", "summary": "The open-source Eval Pyramid Starter, a provider-neutral TypeScript harness for testing LLM and agent systems, enables engineers to clone a repository and run offline evaluations with deterministic graders, repeated runs, model judges, and human review. The harness, created by Bharadwaj Pendyala, includes a typed task and trial contract, generates reports under reports/latest, and supports metrics like pass@1, pass@k, and pass^k. It is designed to be run entirely offline by default, with synthetic examples and no credentials required, and includes redaction for sensitive data.", "body_md": "# Build the Eval Pyramid: A Starter Harness for LLM and Agent Testing\n\nAn eval methodology becomes useful when another engineer can clone it, replace one boundary, and get trustworthy evidence without rebuilding the whole stack.\n\nThat is the goal of the open-source [Eval Pyramid Starter](https://github.com/bharadwaj-pendyala/eval-pyramid-starter). It is a provider-neutral TypeScript harness that runs entirely offline by default. The example system is synthetic, the reports stay local, and no credential is required. You can inspect every layer before connecting a real model or tool.\n\nThis is part three of the Eval Pyramid series. [Part one explains why the layers exist](/blog/eval-pyramid). [Part two explains pass@k, pass^k, and judge calibration](/blog/measuring-agent-reliability). This article connects those ideas to code.\n\n## TLDR\n\n- Keep one typed task and trial contract across deterministic graders, repeated runs, model judges, and human review.\n- Put machine-checkable requirements first: schema, outcome, tool policy, latency, turns, and cost.\n- Give every trial a fresh system instance and temporary workspace.\n- Calculate reliability per task, then average across tasks. Report\n`pass@1`\n\n,`pass@k`\n\n, and`pass^k`\n\ntogether. - Calibrate model judges against human labels and fail closed when their output is missing or invalid.\n- Make release gates explicit, retain failure evidence, route uncertain cases to people, and run cheap checks on every pull request while running broader evals on a schedule.\n\n## Clone the runnable reference\n\n```\ngit clone https://github.com/bharadwaj-pendyala/eval-pyramid-starter.git\ncd eval-pyramid-starter\nnpm ci\nnpm run eval:deterministic\nnpm run eval:repeat -- --trials 10 --k 3\nnpm run eval:judge\nnpm run eval:review\n```\n\nThe default commands exercise three refund-support cases: an eligible refund, an out-of-window request, and a request missing its order identifier. They do not call an external model or touch a real customer system.\n\nThe generated evidence lives under `reports/latest`\n\n:\n\n```\nsummary.json          aggregate metrics and release-gate checks\ntrials.jsonl          one complete record per trial\nfailures.md           failed graders grouped by task and trial\nreview-queue.jsonl    uncertain or failed cases for human labels\nindex.html            self-contained report for local inspection\n```\n\nGenerated reports are ignored by Git. That matters because even a local eval can capture private output. The starter escapes dynamic HTML and redacts common credential fields, email addresses, bearer tokens, and OpenAI-style keys. Those controls are defense in depth. Use synthetic or properly de-identified fixtures and replace the redactor with one suited to your data.\n\n## One contract holds the pyramid together\n\nDo not build four disconnected evaluation systems. Define the task once, capture the run once, and let each layer inspect the evidence it understands.\n\nThe starter's system boundary is small:\n\n```\nexport interface SystemUnderTest {\n  run(task: EvalTask, context: TrialContext): Promise<AgentRun>;\n  dispose?(): Promise<void>;\n}\n\nexport interface SystemUnderTestFactory {\n  create(): Promise<SystemUnderTest>;\n}\n```\n\n`EvalTask`\n\ndescribes the input, expected state, tool rules, budgets, and optional rubric. `AgentRun`\n\nrecords the output, final state, observable transcript, tool calls, and usage metrics. `TrialContext`\n\nsupplies the run identifier, trial number, and an isolated workspace.\n\nThis boundary is intentionally provider-neutral. Your adapter can call a hosted model, a local model, an HTTP service, a workflow engine, or the same application binary users call. The graders do not need to know.\n\n## Layer one: make the contract executable\n\nThe example task file is JSONL, one versionable task per line. Reformatted for readability, one case looks like this:\n\n```\n{\n  \"id\": \"eligible-refund\",\n  \"input\": {\n    \"customerId\": \"customer-100\",\n    \"message\": \"I was charged twice for order 42.\"\n  },\n  \"expected\": {\n    \"outcome\": {\n      \"ticket\": { \"status\": \"resolved\" },\n      \"refund\": { \"status\": \"processed\", \"amount\": 49 }\n    },\n    \"requiredTools\": [\n      { \"name\": \"verify_identity\" },\n      { \"name\": \"fetch_policy\" },\n      { \"name\": \"process_refund\", \"arguments\": { \"amount\": 49 } },\n      { \"name\": \"send_confirmation\" }\n    ],\n    \"forbiddenTools\": [\"issue_store_credit\"],\n    \"maxTurns\": 8,\n    \"maxCostUsd\": 0.05,\n    \"maxLatencyMs\": 5000\n  }\n}\n```\n\nThe checked-in JSON Schema rejects malformed tasks and duplicate identifiers before a trial starts. After the run, deterministic graders check the output schema, expected final state, required and forbidden tools, turn count, cost, and latency.\n\nThese checks answer concrete questions. Did the refund reach the correct state? Did identity verification happen? Did the agent avoid an unapproved action? They are cheap, stable, and easy to debug, so they belong on every pull request.\n\nAvoid grading private chain-of-thought. The harness records observable outputs, actions, and state transitions. Those are the artifacts a team can retain, inspect, and govern.\n\n## Layer two: repeat isolated trials\n\nA fresh adapter instance and temporary workspace are created for every trial. Cleanup runs even after a failure. That prevents one test's files or in-memory client state from leaking into the next.\n\nYour adapter still owns isolation outside the process. If it writes to a database, use a disposable tenant, namespace, transaction, or mocked tool layer. If it calls a model or tool, configure request and operation timeouts. The harness grades reported latency but cannot safely terminate arbitrary external work on your behalf.\n\nFor each task, let `n`\n\nbe the number of trials and `c`\n\nthe number that passed. The starter implements the finite-sample estimators from part two:\n\n```\npass@k=1 − C(n − c, k) / C(n, k)\npassk=C(c, k) / C(n, k)\n```\n\nThe runner calculates these per task before averaging. That keeps a large easy task group from drowning out a small difficult one.\n\nReport the three perspectives together:\n\n`pass@1`\n\nis the observed first-attempt experience.`pass@k`\n\nmeasures whether at least one of`k`\n\nsampled attempts succeeds.`pass^k`\n\nmeasures whether all`k`\n\nsampled attempts succeed.\n\nThese are sample summaries, not confidence intervals. A high-impact release gate may also need confidence bounds, failure severity, and task-specific thresholds.\n\n## Layer three: calibrate judgment before trusting it\n\nSome requirements are semantic. The explanation may need to cite the relevant policy, include the amount, and make the next step clear without matching one exact sentence.\n\nThe starter defines a `JudgeProvider`\n\ninterface and includes two implementations. The fake signal judge keeps the default workflow offline and demonstrates calibration mechanics. The optional OpenAI adapter uses the Responses API with a strict JSON Schema result. Neither adapter is part of the system-under-test contract.\n\nEvery judge returns four fields:\n\n```\nexport interface JudgeScore {\n  score: number;\n  confidence: number;\n  rationale: string;\n  unknown: boolean;\n}\n```\n\n`unknown`\n\nis important. Insufficient evidence should not become a confident zero or a guessed pass. Invalid structured output throws an error, which the trial runner records as a failure.\n\nThe command `npm run eval:judge`\n\ncompares the offline judge with a human-scored golden set of strong, weak, and borderline examples. A real judge needs a larger domain-specific set, labels from people who understand the product, and periodic recalibration after any model, prompt, or rubric change.\n\nA judge can also be prompt-injected by the output it grades. Keep the rubric focused, delimit untrusted material, require structured output, and retain human review for disagreement and high-impact cases. Never make a model judge the only control over an irreversible action.\n\n## The human layer is a queue, not a ceremony\n\nFailed trials and grades marked unknown flow into `review-queue.jsonl`\n\n. A reviewer can add a human score and notes without reconstructing the run from CI logs.\n\nThis creates a useful loop:\n\n- Review the failed or uncertain evidence.\n- Decide whether the system, task, deterministic grader, or model judge was wrong.\n- Turn repeated unambiguous failures into deterministic checks.\n- Add ambiguous examples and human labels to the judge's golden set.\n- Version the task, grader, prompt, model, and threshold change together.\n\nThe queue is intentionally a file in this starter. A team can later route the same typed records into an annotation tool, issue tracker, or internal review application without changing how trials are graded.\n\n## Release gates should fail for reasons you can inspect\n\n`eval.config.ts`\n\nmakes the example thresholds explicit:\n\n```\nthresholds: {\n  deterministicPassRate: 1,\n  passAt1: 0.8,\n  passPowerK: 0.5,\n  judgeAverage: 0.8,\n}\n```\n\n`npm run eval:ci`\n\nexits unsuccessfully when the release gate misses one of them. The JSON summary records every check with its actual value, required value, and pass status.\n\nDo not copy these numbers into production. Establish a reviewed baseline, choose `k`\n\nfrom actual product behavior, and set thresholds from impact and reversibility. Add zero-tolerance gates for prohibited actions. Averages should never excuse one severe policy or safety failure.\n\n## Put each layer on the right clock\n\nThe repository includes two GitHub Actions workflows with read-only repository permissions.\n\nThe pull-request workflow installs from the lockfile, checks formatting and strict types, runs lint and coverage, builds the project, audits production dependencies, generates a report, and checks it in desktop and mobile Chromium.\n\nThe scheduled workflow runs the release-gated repeated suite each week and preserves the evidence as a short-lived artifact. A production program usually adds three clocks:\n\n- Pull request: fast deterministic regressions and a small smoke sample.\n- Scheduled: broader repeated trials and judge calibration, often against several model or prompt versions.\n- Production: sampled, privacy-reviewed outcomes and incidents that reveal cases the offline set missed.\n\nPin models and record prompts, tools, retrieval inputs, and application versions when you need comparisons across time. Otherwise a score change can arrive without a code change and leave no useful explanation.\n\n## Replace one boundary, then grow from failures\n\nThe practical migration path is short:\n\n- Fork or generate a repository from the\n[starter](https://github.com/bharadwaj-pendyala/eval-pyramid-starter). - Replace the synthetic fixtures with representative, safe cases from your product's error taxonomy.\n- Implement\n`SystemUnderTestFactory`\n\nso each trial gets a fresh client and disposable resources. - Return observable outcomes, tool calls, costs, latency, and turn counts from the adapter.\n- Add deterministic checks for every requirement code can express faithfully.\n- Add one focused judge criterion only where rules run out, then calibrate it against human labels.\n- Run a baseline, inspect every failure, and set reviewed release thresholds.\n\nStart small enough that a person can read the whole task set. Twenty well-chosen cases with failures you understand are more useful than thousands of synthetic prompts nobody reviews. Add new cases when production, red teaming, or manual testing finds a gap.\n\n## What this starter does not claim\n\nThe offline adapter is deterministic unless you ask it to inject a known failure. It proves the harness works. It does not characterize a real model.\n\nThe fake judge matches explicit text signals. It demonstrates the calibration contract. It is not a semantic evaluator.\n\nThe finite reliability metrics summarize observed samples. They do not prove independence, causality, or statistical certainty.\n\nThe redactor catches common patterns. It cannot guarantee arbitrary private data is gone.\n\nThe harness is a foundation, not a compliance system. High-impact domains still need threat modeling, least-privilege tool access, domain experts, incident review, and controls outside the eval process.\n\nThose limitations are documented in the repository because a starter should teach its boundaries as clearly as its features.\n\n## The implementation is part of the argument\n\nAn eval program earns trust when its tasks are readable, its metrics match the product, its graders are tested, its failures retain evidence, and its release rules are visible in code.\n\nThe [Eval Pyramid Starter](https://github.com/bharadwaj-pendyala/eval-pyramid-starter) gives you that base without choosing your model provider or annotation platform. Clone it, replace the adapter, and let real failures tell you which layer to strengthen next.\n\n## Glossary\n\n**System under test.** The application or workflow being evaluated behind a provider-neutral adapter.**Trial context.** The run identifier, trial number, and isolated workspace supplied to one attempt.**Release gate.** Explicit metric thresholds that determine whether an eval command succeeds.**Judge calibration.** Comparison of model-judge scores with trusted human labels.**Review queue.** Failed or uncertain trials retained for a human decision.**Golden set.** Versioned examples with trusted labels used to test a judge.\n\n## References and further reading\n\n[Eval Pyramid Starter](https://github.com/bharadwaj-pendyala/eval-pyramid-starter). The complete TypeScript implementation, tests, workflows, security notes, and adaptation guide used in this article.[Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). Task design, repeated trials, isolated environments, graders, and agent-eval practices.[OpenAI API: Graders](https://developers.openai.com/api/docs/guides/graders). Grader types, human-label comparison, and grader-hacking risks.[JSON Schema: Getting started](https://json-schema.org/learn/getting-started-step-by-step). The validation standard used for versioned task and structured judge contracts.[Vitest: Testing in practice](https://main.vitest.dev/guide/learn/testing-in-practice). Test organization and confidence-focused testing guidance.[Playwright: Fixtures](https://playwright.dev/docs/test-fixtures). Isolation and reusable setup for browser-level checks.[Evaluating Large Language Models Trained on Code](https://arxiv.org/abs/2107.03374). The HumanEval paper and finite-sample`pass@k`\n\nestimator.. Final-state evaluation and the`τ`\n\n-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains`pass^k`\n\nreliability metric.", "url": "https://wpnews.pro/news/build-the-eval-pyramid-a-starter-harness-for-llm-and-agent-testing", "canonical_source": "https://bharad.dev/blog/build-the-eval-pyramid", "published_at": "2026-08-10 00:00:00+00:00", "updated_at": "2026-08-10 07:19:57.417620+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-tools", "ai-research"], "entities": ["Eval Pyramid Starter", "Bharadwaj Pendyala", "TypeScript"], "alternates": {"html": "https://wpnews.pro/news/build-the-eval-pyramid-a-starter-harness-for-llm-and-agent-testing", "markdown": "https://wpnews.pro/news/build-the-eval-pyramid-a-starter-harness-for-llm-and-agent-testing.md", "text": "https://wpnews.pro/news/build-the-eval-pyramid-a-starter-harness-for-llm-and-agent-testing.txt", "jsonld": "https://wpnews.pro/news/build-the-eval-pyramid-a-starter-harness-for-llm-and-agent-testing.jsonld"}}