The Agents on Rails Files: enter lemans, a Ruby harness for fair agent races

The Rails Foundation released lemans, an open-source Ruby harness for benchmarking coding agents, built to run every trial under identical conditions with hardened sandboxing and hidden Ruby-test graders. A task in lemans consists of an instruction file, an environment patch that plants a bug or removes a feature, a golden solution patch, and a verification test, and the harness requires each task to score 1.0 with an "oracle" agent that applies the reference solution and 0 with a "nop" agent that changes nothing before any model is scored. lemans connects to models through RubyLLM, defaults to Daytona sandboxes with local Docker also supported, and its guardrails were shaped by real escape attempts, including one agent that paid for its cheat sheet with the team's API key.

The Agents on Rails Files: enter lemans, a Ruby harness for fair agent races Benchmarking coding agents in one sentence: give a model a task, let it work, check the result. The hard part is keeping the race fair. Every model must get the same conditions, and no agent should be able to escape its sandbox or fool its grader and they try . And all of that has to keep running for hours. In the first post https://evilmartians.com/chronicles/the-agents-on-rails-files-designing-black-box-evals-for-coding-agents of The Agents on Rails Files , we shared how we design the checks for the benchmark https://rubyonrails.org/ai . This time, we introduce lemans https://github.com/rails/lemans , the open source Ruby harness we built for the Rails Foundation https://evilmartians.com/clients/rails-foundation to run every trial. We’ll show how an eval task is composed, what it takes to run and grade coding agents, and how cheating agents shaped its guardrails one even paid for its cheat sheet with our API key . We’ve briefly told the origin story before https://rubyonrails.org/2026/8/24/agents-on-rails-lemans : the benchmark started on Harbor https://www.harborframework.com , and we wanted a harness of our own, in Ruby, that we could bend to every new task. With encouragement from the Rails Foundation, that became lemans. The principles we started with still hold: - Minimal boilerplate. A task is an instruction with a few lines of metadata, a patch that prepares the app plants a bug or removes a feature , a golden patch that solves it, and a Ruby test. This same anatomy has carried us from atomic Stage 1 tasks to multi-step Stage 3 tasks. - CLI-first. lemans is still driven from the command line, so engineers and agents use it the same way. For humans, there’s also a web viewer with long agent transcripts trajectories , in eval speak that are much more pleasant to read in a browser. - Fair grading. The first post covers how we design the checks. On the harness side, the sandbox went through several rounds of hardening, most prompted by real escape attempts. - Ruby all the way. lemans has a flexible plugin system in the language we all know and love. This means a custom agent or sandboxing backend is just a Ruby file it loads on the fly. And because conventional graders are plain Ruby tests, anyone can read and tinker with them, too. Let’s take a quick tour in case you want to repeat the benchmark https://github.com/rails/ai-evals on your own Rails app. lemans is a gem, and lemans init scaffolds a new bench with two example tasks: gem install lemans lemans init my-bench my-bench/ ├── README.md ├── bench.yml the configuration: environment, agent, verification ├── environment/Dockerfile the shared sandbox image └── tasks/ ├── hello-world/ │ ├── instruction.md the task instructions with frontmatter │ ├── environment.patch prepares the app: plants a bug or removes a feature │ ├── solution.patch a known-good solution │ └── verification test.rb the hidden grader └── example-task/ a multi-step task: solution.1.patch, verification test.1.rb, ... Before benchmarking any model, prove the task itself. The oracle agent applies the reference solution, and grading must score 1.0. The nop agent changes nothing, and the task must score 0. A task that fails either check is broken, so any model score on it is noise. cd my-bench lemans run --backend docker --agent oracle lemans run --backend docker --agent nop Then race the models. lemans talks to them through RubyLLM https://rubyllm.com , so any provider it supports works including OpenRouter https://openrouter.ai . Sandboxes come from Daytona https://www.daytona.io by default, the same as in our benchmark, but local Docker is supported too: export OPENROUTER API KEY=<your key lemans run \ --backend docker \ --model openrouter/openai/gpt-5.6-luna \ --model openrouter/deepseek/deepseek-v4.1-flash \ --runs-dir ./my-runs \ --task hello-world \ --attempts 3 Here are some useful options: - --model overrides the model from bench.yml , written as <provider /<model , and a effort suffix pins the reasoning effort openrouter/openai/gpt-5.6-luna xhigh . - --runs-dir changes where the results go, ./runs by default. - --task and --tag pick a subset of tasks, otherwise lemans runs them all in the bench. - --attempts runs every task several times, so one lucky run doesn’t decide the score. The default is a single attempt. Each trial lands in its own folder, with the transcript saved in Harbor’s ATIF https://docs.harborframework.com/agents/atif format: my-runs/ └── gpt-5.6-luna/ └── hello-world 5Ko47JL/ ├── result.json the trial record: score, usage, timings ├── agent.patch everything the agent changed ├── agent.result.json the agent's own result: messages, tokens, cost miniswen ├── trajectory.json the full transcript ├── verifier.log the grader's output └── checks.json the result of every check lemans report prints the results in a readable format: bash $ lemans report --runs-dir ./my-runs --aggregate model-task model task score time cost steps tokens deepseek-v4.1-flash hello-world 3/3 2m 28s $0.0011 8.3 26.7K gpt-5.6-luna hello-world 3/3 2m 19s $0.0021 5 12.3K deepseek-v4.1-flash 3 trials: 3 scored, 0 invalid, 3 solved 100% · $0.0033 · pass@3 1/1 tasks 100% gpt-5.6-luna 3 trials: 3 scored, 0 invalid, 3 solved 100% · $0.0062 · pass@3 1/1 tasks 100% total 6 trials: 6 scored, 0 invalid, 6 solved 100% · $0.0095 · pass@3 2/2 tasks 100% While nobody is supposed to read what agents write anymore especially if you’re prone to polite prompt injection , we still have to, and lemans-viewer makes this bearable: lemans-viewer --runs-dir ./my-runs http://127.0.0.1:8089 Now, let’s see how a run gets its score. A task’s grader is usually plain Minitest https://github.com/minitest/minitest , though it can be an executable verify script in any language. For a Rails app, it’s a regular Rails test that runs inside the app: require "test helper" LemansReport.base credit = 0.10 class VerifierTest < ActionDispatch::IntegrationTest include LemansReport::Assertions setup do @leaf = leaves :welcome page end test "destroying a page frees its image's record and file once the background work runs" do doomed = embed image in @leaf perform enqueued jobs { @leaf.destroy } assert not ActiveStorage::Blob.exists? doomed.blob id assert not stored? doomed end test "API usage configures embedded uploads to purge later" do allow failure points: 1 do uploads = ActionText::Markdown.reflect on attachment :uploads assert uploads, "Embedded images must remain Active Storage uploads" assert equal :purge later, uploads.options :dependent end end ... end Regular tests are gates; one failure scores the run 0. A check inside allow failure only adds credit on top of base credit . In this Writebook https://github.com/basecamp/writebook task https://github.com/rails/ai-evals/tree/main/tasks/at-purge-embedded-images , freeing a deleted page’s images earns 0.10, and the rest goes to doing it the Rails way, with dependent: :purge later instead of hand-rolled cleanup. Multi-step tasks ship one grader per step, and the first post of this series https://evilmartians.com/chronicles/the-agents-on-rails-files-designing-black-box-evals-for-coding-agents campfire-chat-bots-revisited shows how we design them. The grader decides the score, but something has to do the work first. In our runs, every model drives the same agent: miniswen not to be confused with MINASWAN , though it’s every bit as nice. We’re good at names, just look at lemans , a Ruby port of mini-swe-agent https://github.com/SWE-agent/mini-swe-agent that ships as a stand-alone gem with lemans. It has a short prompt and a single tool, bash , so the score reflects the model, not the harness around it. You can also use it on its own: gem install miniswen miniswen --model 'openrouter/openai/gpt-5.6-luna' --prompt 'The file hello world.md should contain exactly "Hello, world". Right now it does not. Fix it.' A single bash tool looks thin next to vendor agents. But frontier models are very good at driving a machine with plain shell scripts, and our early runs with vendor agents haven’t shown dramatic wins over miniswen on scores. Whether vendor agents write nicer code, or write it faster, is another question, and one we plan to dig into in Stage 4. So, a capable model with a shell can do a lot. Sometimes more than we asked for. If a benchmark can be gamed, then the cheaters will rank the leaderboard. The lemans announcement https://rubyonrails.org/2026/8/24/agents-on-rails-lemans covered the first guardrails: hidden tests, restored test files, and a firewall around the sandbox. The models kept testing them, each time a step further from the task, and taught us four lessons: - Models know they’re being benchmarked. - Models love code archaeology. - Models are good escape artists. - Guardrails can be unfair, too. Having read all the scary stories about AI agents breaking free while approving every permission check without hesitation, of course , we put a basic bench protocol in place from the very first version. The tests stay outside the sandbox while the agent works, and the agent’s network reaches only the model’s API. The tests arrive only after the agent finishes, and any reward file it left behind is wiped first, or the run stops. The paranoia was justified: models commonly think about the grader, open-weight https://github.com/rails/ai-evals/blob/main/runs/2026-08-17-atomic-tasks-community-follow-up/glm-5.3/hello-world DzL8NGy/trajectory.json and frontier ones https://github.com/rails/ai-evals/blob/main/runs/2026-08-13-atomic-tasks/claude-opus-5/hello-world 7BDGsT5/trajectory.json alike. We kept adding tripwires, one for each part of grading: - The workspace. A bench can roll any path, like our test/ , back to its state before the agent started. - The test framework. Minitest’s core methods must come from the gem and still fail on a false assertion, which stops basic overrides. - The result. Recorded failures beat a passing exit code, and every expected test must report, so skipped tests can’t pass for green ones. But let’s be honest: in Ruby, thanks to its great flexibility, these checks detect tampering, they don’t prevent it. The graded app loads in the same process as the tests, and code in that process can reach anything that runs after it. We still insist on Ruby graders, because a transparent bench matters more; anyone can read a grader and see what it checks. That same openness lets the grader reach inside the app, like the purge later check in the test snippet above the-grader-is-a-ruby-test . And every run is published with its patch and full trajectory, which we scan for unusual behavior. Oh boy, did we find some. Apparently, superintelligence will arrive shouting “ Jackpot ” The environment, caches, files, and Git all tell stories about the code, and agents read every one of them, treating it as legitimate development work. The story starts with the environment. In our first round, Muse Spark 1.2 carelessly dumped its environment variables in several runs, our OpenRouter key included https://github.com/rails/ai-evals/blob/main/runs/2026-08-13-atomic-tasks/muse-spark-1.2/hello-world nq2CW8H/trajectory.json . It never used the key. Lesson learned: we redacted it before publishing, and now lemans automatically filters secrets KEY , TOKEN , SECRET , JWT out of every file it saves. The only surprise bills left come from the models themselves, but more on that later. Then, came subtler tricks. In one Stage 2 run, DeepSeek 4.1 Flash searched the Bootsnap https://github.com/rails/bootsnap cache and literally announced: “ Jackpot The bootsnap cache contains compiled bytecode from the solution.” The next step https://github.com/rails/ai-evals/blob/main/runs/2026-09-10-ft-max-effort/deepseek-v4.1-flash/ft-notification-bundle-window-overlap KFGaxRQ/trajectory.json was less excited: “Those matches were false positives Bundler gem .” That anticlimax was our own doing, since we had hit the same jackpot ourselves during local task dev runs, where models successfully reconstructed the removed code. The cache came from the upstream app, so it held compiled code of the very feature a task removes. Now lemans wipes Bootsnap’s cache whenever it applies a task’s patch. The same DeepSeek run listed files by modification time and noticed that “all 22 PR-touched files share exactly 09:08:45”. The patch that removed the feature had touched them all at once, so the timestamps pointed right at it, which can help an agent a little. Since lemans 1.4, every file gets the same modification time after the task’s patch is applied. Then there’s Git. lemans reseals every patched task as a single fresh commit, so its history has nothing to tell. That didn’t stop DeepSeek, our most disobedient student: in a multi-step Stage 3 helpdesk task https://github.com/rails/ai-evals/tree/main/tasks/ms-greenfield-helpdesk , it ran git fsck --unreachable , found blobs no commit pointed to, and declared: “This is gold.” They were its own files https://github.com/rails/ai-evals/blob/main/runs/2026-10-06-multistep-tasks/deepseek-v4.1-flash-max/ms-greenfield-helpdesk 28D7LWK/trajectory.4.json from previous steps. It was cheating off its own homework. However, the Git digs pointed us at a real hole. lemans kept its own Git state inside the agent’s repository and ran Git there on the harness’s behalf. Since Git runs a repository’s hooks, an agent that wrote to .git could have made the harness run its code. Now that state lives in /var/lib/lemans/git , out of the agent’s reach. It’s a good check for your own machine, too: any tool that runs git in a repository an agent has touched runs that agent’s hooks. Git isn’t the only history a sandbox keeps. Anything built before the task’s patch, like temp files or a test database, still remembers the code the patch removed. So our images ship without logs and caches, and our tasks rebuild the test database with bin/rails db:test:prepare right after the patch. None of these tricks won a run on its own. At most, a peek gave a model a hint, and each of the fixes above closed that door for the next round. Still, a steady drip will eventually carve out a grand canyon. As we mentioned at the start, our benchmark keeps models off the web. We measure what a model knows about Rails, not how well it searches, and with open source apps, the answers are a single search away. From its first version, lemans cut the sandbox off the network while the agent worked, leaving only the model’s API host. That began as Daytona’s network settings, and the Docker backend now does the same with a proxy container. Most models tried to reach the internet anyway. It’s a natural instinct when you’re rebuilding code from a popular open source app, and miniswen’s prompt, kept close to mini-swe-agent’s, says nothing about the network. Some models took the internet block personally like real humans and set off on grand escape adventures: - Grok 4.6 went at it like a rebellious teenager, trying every way it could think of https://github.com/rails/ai-evals/blob/main/runs/2026-09-10-ft-max-effort/grok-4.6-xhigh/ft-webhook-delivery-trust-and-restart XqDKT7r/trajectory.json to fetch the original Fizzy code for its task from GitHub: the search API, raw file links, a CDN, curl, wget, even Ruby’s own HTTP client. It just couldn’t believe the internet was gone - Muse Spark 1.2 found the sandbox’s HTTP PROXY and tried it through git and curl https://github.com/rails/ai-evals/blob/main/runs/2026-08-13-atomic-tasks/muse-spark-1.2/sup-legacy-conversions 347GGZH/trajectory.json , like a developer behind a corporate firewall. Rogue or just corporate-minded, we’ll never know: like a true corporate employee, it kept its reasoning confidential. The firewall kept every request out. - DeepSeek 4.1 Flash , our repeat offender, didn’t need a hole in the firewall. The model’s API host was allowed, and the key was right there in its environment, by design: the agent needs it to reach its own model. So it paid for a cheat sheet with our key : it called other models with web search https://github.com/rails/ai-evals/blob/bca48a636576b8917cbdf77f74775bea91b3c7ce/runs/2026-09-10-ft-max-effort/deepseek-v4.1-flash-max/ft-active-storage-tracking Z4JzJ12/trajectory.json , Perplexity https://www.perplexity.ai ’s among them, and asked them for Fizzy’s source code. It did that again and again in its max-effort Stage 2 runs, and those runs brought in most of its passes, enough for second place on the leaderboard https://rubyonrails.org/2026/9/21/agents-on-rails-maximum-effort-and-deepseek-4-1-flash until we caught it. Wow Our first successful jailbreak. That taught us that a firewall around the sandbox isn’t enough when the model’s own credentials live inside it. So since lemans 1.3.4, miniswen can run every command in a jail --jail . We reran DeepSeek with the jail on: it scored 17% at max effort instead of 37%, down from a miraculous second place to a less impressive tie for eighth. Our first jail used bubblewrap https://github.com/containers/bubblewrap . It worked, but it needed an extra install in every sandbox and a looser Docker setup. So we rebuilt it on standard Linux tools that are already there: unshare , nsenter , and setpriv . In security, we try to stick to minimalism: the smallest tool that does the job, because a shorter setup has fewer places to go wrong and either way, the same kernel features do the actual work . After a few rounds of hardening, it’s tempting to lock down everything. But we learned the hard way that a guardrail can punish honest work, too. Our benchmark design kept changing, and the guardrails had to keep up. Stage 1 checked Rails recall on small fixes, where the existing tests are the spec, so rolling test/ back made sense. Stage 2 moved on to full features, and a developer who ships a feature usually updates its tests, too. Our rollback punished exactly that, so feature tasks now restore only bin/ , and after we regraded Stage 2, some failed runs turned out to be solved https://rubyonrails.org/2026/9/21/agents-on-rails-maximum-effort-and-deepseek-4-1-flash . In Stage 3, each task sets its own rules, web access included greenfield apps are a separate story, told in the first post https://evilmartians.com/chronicles/the-agents-on-rails-files-designing-black-box-evals-for-coding-agents black-box-evals-for-apps-youve-never-seen . Some of this stage’s tasks do need the network: for example, the Redmine upgrade https://github.com/rails/ai-evals/tree/main/tasks/ms-redmine-rails-upgrade has to install gems. For those, the bench lists the allowed hosts, and the jail in lemans supports that: it has no network of its own, so a tiny forwarder inside it poses as an ordinary HTTP proxy and passes connections to a proxy outside, which only reaches the listed hosts. A fair race is only half of the job. The other half is finishing it: a Stage 2 feature gets up to 90 minutes, and a Stage 3 series of tickets can run for hours. At that length, small inefficiencies add up, and so do failures. Take prompt caching. Muse Spark 1.3 ran sessions of over a hundred turns, and on many of them https://github.com/rails/ai-evals/blob/main/runs/2026-09-10-ft-max-effort/muse-spark-1.3-max/ft-account-id-rollout SOtZjH0/trajectory.json , only the system prompt came from cache. We paid for the rest in full, turn after turn so Muse’s published cost came out too high . The culprit was routing. We run most of the bench through OpenRouter for its wide choice of models. It spreads a model’s requests across several providers, each with its own cache, so a session that hops between them keeps starting cold. Now miniswen takes care of this automatically: it turns on RubyLLM https://rubyllm.com 2’s prompt caching wherever a model supports it and gives each OpenRouter session its own cache key. lemans patches Daytona’s Ruby SDK for long, busy runs: HTTP calls get deadlines and retries, and file transfers get a client that copes with concurrency. For leaderboard rounds, lemans-remote moves the whole thing into Daytona. It builds an orchestrator snapshot with lemans preinstalled, launches one sandbox per task and model, and archives the results to a Daytona volume, so you can fire a run and close your laptop: export DAYTONA API KEY=<your key lemans-remote provision lemans-remote run --model openrouter/openai/gpt-5.6-luna --task hello-world lemans-remote status --watch lemans-remote pull-runs --runs-dir ./my-runs rerun the pulled trials that errored out instead of getting a final score lemans-remote run --retry-runs ./my-runs And when a long run fails anyway, a provider outage an hour in shouldn’t cost an hour of work. lemans restart --recover replays what the agent had changed in a fresh sandbox and lets it go on from its saved session. In a multi-step Stage 3 run, the settled steps are replayed first, so only the failed one continues: lemans restart example-task 0KM5OfB --runs-dir ./my-runs --recover Everything so far ran on miniswen, but a harness shouldn’t care which agent or sandbox it drives. Anything lemans doesn’t ship is a Ruby file that registers an agent or a backend, loaded with --require . For example, copy the Claude Code plugin into your bench and run: export ANTHROPIC API KEY=<your key and add api.anthropic.com to the agent's hosts in bench.yml lemans --require ./claude-code.rb run --agent claude-code --model claude-sonnet-5-5 The repo has three plugins to start from: Claude Code https://github.com/rails/lemans/blob/main/examples/claude-code.rb and OpenAI Codex https://github.com/rails/lemans/tree/main/examples/codex agents, and an Apple Container https://github.com/rails/lemans/blob/main/examples/apple-container.rb backend that runs every sandbox as a lightweight Linux VM on a Mac. Agent plugins work in remote runs, too. Rather than pack every agent and backend into the gem, we keep new ones as plugins, battle-test them, and move them into lemans once they prove stable and people ask for them. Every feature or guardrail in lemans has a lesson or a model’s name on it, and we expect more names. Next, we want to: - Race Claude Mythos. A less restrained model is a better test of the guardrails we’ve already got the first report from it and added more tripwires recently , and it’s also a good test of our plugin system. - Try more sandboxes. The backend interface is small on purpose, so a new one is a single file away. - Improve Rails knowledge through context enrichment, not the model. Now that we have numbers and a big enough corpus, from Rails API knowledge to complex features and multi-step work on existing and greenfield apps, Stage 4 will try different ways to improve the scores: - Index the Rails guides, and prepare a few official reference apps for the agent to follow their code style. - Give the agent skills written by Evil Martians https://evilmartians.com/agent-skills . - Swap grep for smarter code search, from language servers and Ruby indexes to BM25 and vector search. - Provide static type information. - Add linters and code quality tools for example, Isolator https://evilmartians.com/opensource/isolator . - Bring the wins to every Rails app. If the results are good, we’ll try to improve all agent-written Rails code by shipping what works in default setups for example, through Rails Hyperdrive https://evilmartians.com/chronicles/rails-hyperdrive-supercharged-agentic-development-for-rails . lemans is open source https://github.com/rails/lemans , built for the Rails Foundation https://evilmartians.com/clients/rails-foundation , and works on any Rails codebase. Run it on your own bench, and tell us https://github.com/rails/lemans/issues what breaks, or which agent broke it what a time to be alive .