Ruby on Rails just published the first public, same-harness benchmark of frontier models doing real Rails work. Eight models, 21 atomic tasks, 504 runs, $491. The result is a clean cost-versus-accuracy curve that is immediately useful for anyone pointing an agent at a Rails codebase.
What the benchmark actually tested #
The Rails team used a frozen harness (lemans
), a minimal agent (miniswen
), and one bash tool. Every model saw the same lightweight prompt and the same step/cost limits. Each task was a realistic bug report, security finding, feature request, performance fix, or flaky test on the Writebook codebase. Crucially, the task descriptions never named the Rails API they hinged on - the model had to know it.
Judging was by hidden behavior tests: pass/fail on behavior, not implementation style. API recall was scored separately.
The leaderboard: accuracy and cost #
| Model | Accuracy | Cost notes |
|---|---|---|
| Claude Opus 5 | 92% (58/63) | ~132x Luna; ~$120 total |
| Claude Fable 5 | ~90% default; ~95% possible | Refused one pen-test-style security task | | Kimi K3 | 90% | Half of Fable’s bill | | GPT-5.6 Sol | 84% | ~$0.52/run; ~36x Luna | | Muse Spark 1.2 | ~76% | ~3x Sol (~$98) | | GPT-5.6 Luna | 73% default; 89% at xhigh | $0.91 default; $2.34 xhigh | | GLM 5.2 | 67% | Mid-pack API recall | | DeepSeek V4 Flash 0731 | 65% | 8% API recall |
What stands out #
A dollar gets you most of the way. GPT-5.6 Luna solved 73% of the tasks for $0.91. That is a strong default for low-stakes Rails work.
Every point after that gets expensive. Opus 5 added 19 points over Luna for roughly 132x the cost. The top cluster - Opus, Fable, Kimi K3 - is meaningfully better, but you pay for it.
Reasoning effort scales sharply on Luna. Moving Luna from default to high
jumped it to 86% for $1.36; xhigh
hit 89% for $2.34. That is nearly Fable/Kimi territory for pocket change.
Knowing Rails is what separates the models. API recall ranged from 8% (DeepSeek V4 Flash) to 35%. Runs that used the correct Rails API solved 92% of the time. Hand-rolled fixes still solved 87%. But when a model saw the API and built custom code anyway, success dropped to 64%. The lesson: model choice matters, but prompt/harness design that surfaces the right API matters too.
Safety refusals are a real failure mode. Fable 5 likely hit ~95% but refused a task worded like a penetration-test report. If your agent work touches security, this is worth testing before you commit to a model.
How it compares to other framework benchmarks #
Rails is not the only project asking whether agents can work inside a real codebase. Several related benchmarks exist, though none are a perfect apples-to-apples match:
AppSignal’s Rails vs Django vs Laravel benchmark compared framework-level agent performance across Rails, Django, and Laravel on a set of backend tasks. It is useful for framework choice, but it does not isolate model differences under the same harness. - Martin Alderson’s 19-framework token benchmark measured token throughput and cost across many frameworks and providers. It answers “how fast and cheap is the plumbing,” not “does the model know my framework.” - ABC-Bench is a broader agent coding benchmark with tasks in multiple languages and frameworks. It tests general agentic coding, not framework-specific API recall. - 7minai constraint decay work looks at how model performance falls as task constraints pile up - relevant for long-horizon agent runs. - Reflex full-stack comparison benchmarks end-to-end full-stack agent workflows, closer to product-building than atomic API fixes.
The Rails report is distinctive because it is framework-native: same codebase, same harness, same tasks, with API recall separated from raw accuracy. That makes it a strong complement to general coding leaderboards like SWE-bench.
Where to use this on Tokenstead #
We have folded the Rails results into the model cards for every model in the benchmark. Look for the Agents on Rails benchmark (Aug 2026) callout on the pages for Claude Opus 5, Claude Fable 5, GPT-5.6 Sol, GPT-5.6 Luna, Muse Spark 1.2, Kimi K3, GLM 5.2, and DeepSeek V4 Flash 0731.
If you are choosing an agent harness for Rails work, the agent harnesses directory lists the tools that sit between an LLM and your codebase. The benchmark suggests the harness matters almost as much as the model: a harness that surfaces the right Rails API closes a lot of the gap between a cheap model and an expensive one.
Practical takeaways #
Start with Luna for exploratory work.$1 for 73% is a remarkable baseline. - Pay up only when the task is high-stakes. Opus 5, Fable 5, and Kimi K3 are the safety net for tasks where a missed fix is expensive. - Tune reasoning effort before swapping models. Luna’s xhigh run is a reminder that effort settings are a cheap lever. - Test security tasks specifically. Fable’s refusal shows that safety alignment can work against you on legitimate security work. - Build harnesses that expose Rails APIs. The 92% solve rate when the right API was used is the clearest design signal in the whole report.
The benchmark is live at rubyonrails.org/2026/8/13/agents-on-rails-the-first-benchmark-report.