{"slug": "what-i-learned-building-evaluations-for-coding-agents", "title": "What I Learned Building Evaluations for Coding Agents", "summary": "A developer built ShipTest, an npm-installable evaluation framework released in April 2026, to test coding agents against a team's own pull requests rather than public benchmarks like SWE-bench. ShipTest checks out repositories at the commit before a merged change, runs agents through the open-source PI harness across Git worktrees for concurrent runs, and compares candidate diffs with tool-call and token-usage data. The developer stopped spare-time work on ShipTest after joining an AI Engineering team to evaluate its coding agents using the Mastra framework.", "body_md": "[← All writing](https://dblc.fyi/blog/)\n\n# What I Learned Building Evaluations for Coding Agents\n\nIn April of this year, I set out to build an evaluation framework to move away from “vibe-based” assumptions about models and towards evidence-backed assessments. I wanted to answer a question that none of the existing benchmarks could answer for me:\n\n*“How does this model perform in my codebase?\"*\n\nI spent most of my time making agent runs observable, but less time tackling the hardest problem: evaluating the quality and correctness of their implementations.\n\n## Existing benchmarks\n\nI’d been reading agentic coding benchmarks like [SWE-bench](https://www.swebench.com/) and looking at aggregated results on [Artificial Analysis](https://artificialanalysis.ai/). I understood that higher scores generally meant better performance on a particular benchmark, but that didn’t translate into anything particularly useful for me. Whilst both were helpful for getting a general sense of what models were capable of, they never reflected *my work*.\n\n## Replaying pull requests\n\nThis led me to build [ShipTest](https://github.com/Dava96/ShipTest) in April 2026. The fundamental idea was that you could install an npm package into your codebase and run evaluations against your own code using your own pull requests.\n\nThe principles were simple:\n\n- Agents wouldn’t have previously seen your codebase, reducing the chance that they’d already encountered the problems being evaluated.\n- You know how code should be written in your own codebase because you’ve got firsthand experience building it.\n- The evaluation environment would always be reset, preventing agents from accessing Git history that could reveal the original implementation.\n- The evaluation environment should reflect a real development environment, using an agent harness that developers could actually use.\n\nMy focus was on developer experience. I didn’t want people to fight my framework; I just wanted it to work. You could use AI to generate evaluation tasks from Git history or merged pull requests, or write them by hand.\nThe instructions in the `task.md` file were passed directly to the model, as if the user had typed them themselves.\n\n*An example task prompt in ShipTest.*\n\nThe underlying harness used to orchestrate the agents was [PI](https://pi.dev/). I chose PI because it’s a minimal harness that isn’t proprietary to any of the AI labs.\n\nThis gave me a more consistent environment for comparing different models, whether they were from OpenAI, Anthropic or Google. Rather than using a harness designed specifically for a benchmark, I wanted to use one that reflected how developers could actually work with these models.\n\nTask execution revolved around Git and Git worktrees. I used Git to check out the repository at specific commits, typically the commit immediately before the change I wanted to evaluate was merged. Worktrees allowed me to run multiple evaluations concurrently.\n\nI developed ShipTest to the point where I was using it to evaluate itself. I’d take pull requests I’d previously merged, run different models against the same tasks, and compare their implementations using the built-in pull request viewer, alongside information such as tool calls and token usage.\n\n*The candidate diff view presents a model’s proposed changes alongside its evaluation evidence.*\n\n*The outcome matrix brings model attempts together for comparison.*\n\n## Tackling AI evaluations professionally\n\nAround this time, I interviewed for and secured a position on the AI Engineering team at my current workplace. I was brought onto the team to evaluate our coding agents, with the goal of improving their performance and increasing their impact across the wider tech department. Since I’d now be working on the same problem professionally, I stopped developing ShipTest in my spare time.\n\n## Evaluating agents at work\n\nWe were already using the [Mastra](https://mastra.ai/) framework in a few places, mainly for conversational AI and triaging support tickets. It felt like the natural choice for evaluations because other teams were already familiar with it.\n\nMastra provides functionality and APIs for both evaluations and orchestration, including access to tool calls, token usage and scorers (evaluations).\n\nI put together a small dataset of reproducible tasks that would run against our API, ranging in difficulty from very easy to very hard.\n\nFor the scorers, I defined metrics that I thought would give useful indications of model behaviour. These used a combination of LLM-as-a-judge evaluations (using an LLM to assess another LLM’s output) and deterministic tests.\n\nMastra’s UI is quite information-dense, so I built a custom interface on top of it to make the evaluation results easier to understand. This included a pull request viewer that presented the models’ proposed changes in a familiar format.\n\n## Retrospective\n\nLooking back at both projects, I built the infrastructure needed to run evaluations and observe model behaviour and implementations. However, what both systems lacked was an automated way to verify whether an implementation was actually correct. That’s arguably one of the most difficult parts of evaluating coding agents.\n\nI had a few ideas about how to tackle this, but none of them felt quite right.\n\nOne approach was similar to SWE-bench: using tests from the original merged pull request to validate the model’s implementation.\n\nThe problem is that these tests might be too closely coupled to the original implementation. For example, a test might assert that a particular field has a specific name. If the model used a different name that still satisfied the original requirements, the test would fail despite the implementation potentially being valid.\n\nOne way around this would be to make the task instructions extremely specific about the expected implementation. But I’ve got mixed feelings about that, because you’re essentially nudging the model towards the “approved” answer. That doesn’t necessarily reflect real development work either.\n\nI think a possible answer is to create specific tests for each evaluation task and grade the implementation rather than relying on a binary pass or fail.\n\nInstead of failing the entire evaluation because one field name doesn’t match, you could deduct points from the model’s overall score.\n\nThe caveat is that you’d need some form of weighted scoring system to account for the severity of different problems. A minor naming discrepancy shouldn’t necessarily carry the same penalty as an implementation that fundamentally doesn’t work.\n\nIf I was starting again, I would spend more time working out the requirements and how to evaluate the results the model produced. Instead of spending the time on behavioural observations.", "url": "https://wpnews.pro/news/what-i-learned-building-evaluations-for-coding-agents", "canonical_source": "https://dblc.fyi/blog/a-retrospective-on-ai-evaluations/", "published_at": "2026-11-22 11:01:49+00:00", "updated_at": "2026-11-22 11:11:45.808909+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": ["ShipTest", "PI", "SWE-bench", "Artificial Analysis", "Mastra", "OpenAI", "Anthropic", "Google"], "also_reported_by": [], "takeaway": "Evaluate coding agents by replaying your own merged pull requests from the commit just before the change, resetting the environment so agents cannot read Git history, and running the open-source PI harness across Git worktrees to compare models on identical tasks.", "recommendation": {"advice": "Build evals from your own merged PRs: check out the pre-merge commit, reset the environment to hide Git history, and run each model through the same non-proprietary harness so diffs and token usage are comparable.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/what-i-learned-building-evaluations-for-coding-agents", "markdown": "https://wpnews.pro/news/what-i-learned-building-evaluations-for-coding-agents.md", "text": "https://wpnews.pro/news/what-i-learned-building-evaluations-for-coding-agents.txt", "jsonld": "https://wpnews.pro/news/what-i-learned-building-evaluations-for-coding-agents.jsonld"}}