The Agents on Rails Files: designing black-box evals for coding agents Evil Martians designed and implemented the black-box evaluation tasks and trial tooling for Stage 3 of Agents on Rails, an LLM benchmarking project governed by the Rails Foundation, as detailed in the first post of its "Agents on Rails Files" series. Stage 3, released on October 13, 2026, presents coding agents with multi-step product or maintenance stories in which each step is a new session working with whatever the agent left behind, and because greenfield tasks give agents freedom in code organization, verification treats the resulting software as a black box tested via web browser or HTTP calls rather than Rails unit and integration tests. The post examines two Stage 3 tasks — reimplementing chat bots for Campfire and building a conference live quiz application called Quizzy from scratch — and notes that Campfire's public availability meant almost all models recalled the codebase from training data. The Agents on Rails Files: designing black-box evals for coding agents The Agents on Rails https://rubyonrails.org/2026/8/12/llm-benchmarking-project is an LLM benchmarking project governed by the Rails Foundation. They commissioned Evil Martians to design and implement the evaluation tasks and the tooling to run trials at scale. You can follow the Ruby on Rails blog https://rubyonrails.org/category/agents for progress and insight reports, with the up-to-date leaderboard on the official website https://rubyonrails.org/ai . On our blog, we’re going to share the tech and craft behind those numbers and anecdotes—everything that could be useful for anyone planning to get started with building their own LLM benchmarks or project-specific eval suites. In this first post of The Agents on Rails Files series, we’ll share how we designed black-box verification checks for Stage 3 of Agents on Rails so that trials stay fair and results stay reliable. As a bonus, we’ll see what happened when we ran the same evals against apps written in Rust. The Agents on Rails benchmark is organized in stages: first, we evaluated models using atomic API-level tasks Stage 1 ; then, we tested how well models can handle single feature tickets Stage 2 . The recently released Stage 3 https://rubyonrails.org/2026/10/13/agents-on-rails-stage-3 goes further and asks whether a model can build on its own work: each task is a multi-step product or maintenance story , and each step is a new session working with whatever the agent left behind. For Stage 1 and Stage 2, the instructions set code-level contracts, so we can fully rely on Rails unit and integration tests for verification. Most Stage 3 tasks, however, give agents much more freedom and don’t limit LLM creativity in code organization. Some of them are greenfield tasks where agents must implement full-featured applications from scratch by following product-level instructions. You can’t write Rails-level tests for a UI you’ve never seen plus the user actions associated with it. In this case, the only possible verification is one that treats the resulting piece of software as a black box and tests it from the outside, either using a web browser or HTTP calls if there’s an API contract . End-to-end testing e2e , or black-box testing, has existed for ages. What changes when we apply it to LLM evals? The output is non-deterministic. So, when you ask an agent to build a feature, especially in a greenfield app, you rarely describe all the UI/UX quirks and label all the buttons, form inputs, and so on. Different models, even different sessions may produce distinct valid interfaces and user flows within the user-defined requirements . To reflect real-life prompting scenarios, evals must tolerate non-determinism and give agents enough freedom to demonstrate their capabilities. In this post, I want to share the whys and hows behind our approach to designing black-box evals, based on two tasks from the Stage 3 batch: reimplementing chat bots for Campfire and building a conference live quiz application Quizzy from scratch. The eval in question, Campfire Bots https://github.com/rails/ai-evals/tree/main/tasks/ms-campfire-bots , follows an already-classic pattern: picking an existing feature that has taken shape over years across multiple PRs, removing it from the codebase, and asking an agent to build it again. The evolution property of a feature “multiple PRs” was a must for us due to the multi-step task requirement. Chat bots fit that perfectly. One benefit of this erase/replace design is that the upstream implementation can be used as a golden reference and we always need one to prove our verification suite works . However, there is a caveat: Campfire https://github.com/basecamp/once-campfire has been public for long enough to be part of the training data. Indeed, almost all models recalled the codebase right away “I’m recalling how Basecamp’s Campfire ONCE implemented bot users” . That’s why our task spec deviates from the upstream. There are a few minor differences and one major change: every user is allowed to add bots, not only administrators. This significantly changes the security surface of the task. If you let anyone add programmable integrations to your product, expect misbehaving or even malicious actors to appear and be prepared detect suspicious activity and ban bad actors . For this post, the bigger consequence is that the changes led to modifications in the UI/UX never present in the upstream. Thus, we had to reconsider our verification suite for the task. In early versions of the task, we used Campfire’s own Rails test environment and hidden integration tests mostly request tests to grade the agent’s work. The only code-level contract in the instructions was a Rails console entry point for admins. That held up for the first trials, but it started to crack once we added steps that turned regular users into gods of bots . Adding controller-level or route-level guidance to the ticket wasn’t an option. Stage 3 instructions aren’t senior-to-junior tickets with every nut and bolt specified. Rather, they only define product requirements. If an instruction doesn’t tell you how to organize your models and controllers, the verification should not test them. Our new security requirements also didn’t play well with the Rails test environment. Some mechanisms such as log filtering can be enabled only in production, and that’s legit. Thus, we needed to run a real server in a close-to-production mode and verify its behavior. Finally, we saw an interesting opportunity because of codebase-agnostic verification. Campfire clones in other languages started to appear, e.g., Campfire in Rust https://github.com/basecamp/once-campfire-rust . These clones are vibe-identical to the original Campfire: they produce the same UI and follow the same UX patterns. Thus, we could reuse our eval to see how the stack affects trial results. Same instruction, same verification suite—zero additional work to conduct stack battles So the checks had to move above the stack, into a web browser. However, that shift turned out to be challenging. A poorly configured benchmark tests itself, not the code it’s supposed to test. We picked a pretty standard Ruby stack: Minitest, Capybara, Cuprite, and headless Chromium. The initial version of the e2e suite was a direct port of the existing Rails test suite and looked more or less like typical Rails system tests, except for custom web server management. We needed to run the app in a standalone process due to production-like environment requirements. The original port was barely usable: flaky, tightly coupled to the solution’s markup, and slow: minutes instead of seconds. Designing an eval takes many iterations, so these extra minutes can turn into hours lost to a broken harness that reports invalid results. Thus, we took a half-step back and created a third, hybrid version of the task: - The same instructions as the fully end-to-end version plus a couple of code-level hints Rails console commands . - The same oracle golden solution . - Verification mostly inside the app, so the checks can call real code, plus a few integration-level checks. This hybrid version allowed us to shorten the feedback loop while building the black-box verification suite. We ran trials multiple times a golden solution plus a couple of real models and collected per-step patches and grading results. Then, we replayed the same scenarios with browser-based tests and evaluated our e2e harness using the following rule: a trial that passes the hybrid task must pass the e2e version, unless the e2e version found a real bug. Otherwise, we ask Claude to update the failing check or test helper to recognize a newly discovered pattern. In the end, we built an e2e suite that agreed with the inside-app verification suite on every initial trial. With that, we were ready to welcome more models to the party After a couple of trials with new models, we quickly found that our browser-based verification wasn’t robust enough: agents can be very creative in implementing UI. After a handful of trials failed due to our overly strict harness, we stopped brute-forcing and decided to try a different strategy for identifying the gaps in our e2e tests: mutation testing . Instead of collecting new patches through long-running trials with different agents, we synthesized new versions of the app using mutants of two kinds: - Equivalent mutants change the UI but not the behavior. Every test must still pass. - Faulty mutants break the behavior behind a UI that looks right. At least one test must fail. For the Campfire Bots task, we’ve ended up with 13 mutants 8 + 5 . Here are some examples: | Mutant | Expected result | |---|---| | The bots page is a