Another week, another step for Agents on Rails. This one is a big one: lemans, the harness behind every number we’ve published, is now open source. We also ran four new models: Sonnet 5, Terra, an open-weight Qwen you can run on your own machine, and one that won’t tell us its name.
If you’re here for the ride, read on for the tour of lemans. If you’re here for the scores, they’re on the leaderboard, and the raw runs are updated too.
Qwen 3.8-27B is the one we (and many of you) were most curious about: it’s open weight and small enough to run on hardware you may already own. Should you ditch your cloud LLM subscriptions and go #supportlocal
? Well: it scores 48 of 63, right in the middle of the pack. Pretty good for a model that you can run locally. However, there’s a catch! Qwen takes a median of 27 minutes per run (we even had to raise our timeouts from 30 to 60 minutes to fit it), which is 1.7x the next-slowest model in the benchmark. Its Rails API recall is also the worst at 7.9%. Impressive for a model that might be run on a laptop.
Terra is another great option from the GPT family: it scores 49 of 63 for 20 cents a run, at a median of 182 seconds, which means it is now the fastest in the entire field bumping Luna into second place.
OpenAI’s three models (Sol, Terra, and Luna) score 84%, 78%, and 73%, in exactly that order by price. The more you pay, the better you do, and all three run in the fastest third of the field, so the stronger model never costs you time.
Anthropic’s lineup is the opposite story. Sonnet 5 lands at 44 of 63, the weakest Anthropic result we’ve recorded, stretching the family’s range from 92% down to 70%. Wide, and hard to explain. Sonnet 5 reaches for the right Rails API more often than Opus 4.8 (25.4% against 15.9%). It still finishes six runs behind, at twice the time. Better recall doesn’t buy the result.
ox-alpha is the odd one out. It’s a stealth model on OpenRouter: unnamed, unattributed, and free while it’s in preview. It scores 52 of 63, matching Grok 4.6 exactly, with 28.6% Rails API recall.
We started our Agents on Rails experiments using Harbor, and it worked okay-ish. One thing bothered us from the very first day of this journey: why do we have to use this Python-esque (in all senses) good-enough tool? Why does our beloved Ruby and Rails community have no such tooling? We believed we deserved it. So with further encouragement from the Rails Foundation, we wrote one: lemans.
We didn’t build lemans just to stay 100% Ruby and to be able to tweak the harness and tasks however we like (thanks to Ruby’s openness). We also wanted:
You can see all of this yourself by giving lemans a try.
Clone the evals repo, install the lemans
gem, grab your LLM provider and Daytona API keys, and you’re ready to rock!
Which LLM? Any supported by RubyLLM, the workhorse of our harness’s LLM integration. Provide a token of your choice via environment variables, e.g., ANTHROPIC_API_KEY=<token> lemans run ...
or OPENROUTER_API_KEY=<token> lemans run ...
.
Daytona is the primary sandbox provider supported by lemans. We’ve also just shipped a Docker backend, so you can run everything on your machine—useful for developing and debugging new tasks. However, in this post we will stick to Daytona, since this is what allowed us to spin up dozens of isolated concurrent trials for the Agents on Rails page.
We’ve figured out the dependencies. Now let’s run some tests. Here is a complete snippet:
git clone https://github.com/rails/ai-evals.git
cd ai-evals
gem install lemans
export DAYTONA_API_KEY=<your key> OPENROUTER_API_KEY=<your key>
lemans run \
--model openrouter/openai/gpt-5.6-luna \
--runs-dir ./my-runs \
--task ac-throttle-search \
--task hw-scoped-broadcast \
--attempts 3
The lemans run
command takes a bunch of arguments. The most useful ones are:
--model
: specify the model on the fly (overriding the one defined in bench.yml
); we use the following notation: <provider-id>/<model-name>
--task
: pick the tasks you want to run (all tasks are executed by default)--runs-dir
: choose the folder to store the results (could be omitted but the default, ./runs
, is already populated with our historic results)That’s it. When the run finishes, you can find the results stored in the my-runs/
directory:
my-runs/
gpt-5.6-luna/
ac-throttle-search__aGApA4O/
agent.patch
agent.trajectory.json
checks.json
result.json
verifier.log
...
Each trial is represented by a single folder with the result record (result.json
) and other evidence (for example, a full agent session log called trajectory in ATIF format).
We don’t expect you to read these files as-is—there is the lemans report
command:
$ lemans report --runs-dir ./my-runs
task agent model reward outcome cost_usd steps tokens duration trial
ac-throttle-search miniswen gpt-5.6-luna 1 completed 0.0131 12 153536 130.7 ac-throttle-search__UDIqyvU
ac-throttle-search miniswen gpt-5.6-luna 1 completed 0.0122 12 139615 155.5 ac-throttle-search__OVrDYDN
ac-throttle-search miniswen gpt-5.6-luna 1 completed 0.0159 16 236046 191.6 ac-throttle-search__NOQ0EF0
hw-scoped-broadcast miniswen gpt-5.6-luna 0 completed 0.0153 17 220508 516.8 hw-scoped-broadcast__EdajQdT
hw-scoped-broadcast miniswen gpt-5.6-luna 1 completed 0.013 16 180751 137 hw-scoped-broadcast__Lpx2qrc
hw-scoped-broadcast miniswen gpt-5.6-luna 1 completed 0.0133 15 185271 136 hw-scoped-broadcast__vBdxmk9
6 trials: 6 scored, 0 invalid, 5 solved (83%) · $0.0828 · pass@3 2/2 tasks (100%)
$ lemans report --runs-dir ./my-runs -A model-task
model task score time cost steps tokens
gpt-5.6-luna ac-throttle-search 3/3 2m 36s $0.0138 13.3 176399
gpt-5.6-luna hw-scoped-broadcast 2/3 2m 17s $0.0138 16 195510
6 trials: 6 scored, 0 invalid, 5 solved (83%) · $0.0828 · pass@3 2/2 tasks (100%)
This is exactly how we obtained the results published on the Agents on Rails AI page: many lemans run
-s and many lemans report
-s.
lemans also ships the lemans-remote
CLI to schedule task execution on the Daytona platform (so you can launch a bunch of tasks and pull the results as they’re done).
We designed lemans not just as a CLI but as a library you can use in your applications (and in a potential future lemans-web
). The library has a component-based architecture, meaning that you can easily plug in custom backends, storage providers (to store results anywhere), and concurrent executors.
Finally, lemans also includes a companion gem/CLI, miniswen. miniswen is a Ruby port of mini-SWE-agent, which can also be used as a dead-simple standalone coding agent:
gem install miniswen
export OPENROUTER_API_KEY=<your key>
miniswen \
--model=openrouter/openai/gpt-5.6-luna \
-p "Create a new eval task that would require an LLM to create a complex migration that wouldn't cause downtimes"
So far, we’ve talked only about how to use lemans. Now, let’s peek under its hood and learn about the tricks we’ve implemented to make it even more useful.
A benchmark is worth nothing if an agent under eval can tamper with the tests, make up evidence or fool you some other way. We learned this the hard way, and turned these learnings into the harness’s guardrails. lemans comes with many; the following three deserve your special attention.
Every task ships a hidden verification_test.rb
the agent never sees. However, hiding just a test file is not enough. We also run the bin/rails test
command before running the eval as a sanity check. An agent that can edit test/
can make the existing suite green without fixing anything. Worse, an agent can rewrite the bin/rails
executable itself and make not only the application tests but also the verification_test.rb
turn green without doing anything else!
So before the verification phase, lemans restores the graded surfaces (test/
, bin/
, and config/environments/test.rb
by default) from a snapshot taken before the agent started: any changes to these files disappear. What’s left is the app code judged by tests the agent never had access to.
Daytona sandboxes get network access during setup: we need to execute gem install miniswen
. We could be stricter and allow access only to rubygems.org
, but such a complication wouldn’t make sense: we can simply turn the firewall on as soon as the agent starts working. That’s exactly what we do: limit network access to the LLM provider’s API host and nothing else. The agent can’t fetch a patch from GitHub, can’t look up the task on the web, and can’t use a phone-a-friend lifeline. The agent is left alone with its inner Rails knowledge.
As you’ve already learned, lemans stores a bunch of files on disk for each run (and we publish them on GitHub). What could go wrong? Well…
You may have seen that in the last round, Muse Spark 1.2 opened many of its runs by dumping env for a look around, which put our OpenRouter key straight into the trajectory files. Luckily, we caught and redacted it before publishing. We turned this almost-incident into a feature: lemans now filters the provider credentials it knows about out of every file it writes to disk, just in case there is some evil agent scanning your file system. So we have Muse Spark 1.2 to thank for that feature.
What’s next for lemans?
We will probably be tuning it here and there while preparing the next set of tasks, but most features are already there, waiting for your feedback. Please give it a try and let us know what you think!