# The first Rails agent benchmark: what it means for picking a model > Source: > Published: 2026-08-13 18:34:03+00:00 **Ruby on Rails just published the first public, same-harness benchmark of frontier models doing real Rails work.** Eight models, 21 atomic tasks, 504 runs, $491. The result is a clean cost-versus-accuracy curve that is immediately useful for anyone pointing an agent at a Rails codebase. ## What the benchmark actually tested The Rails team used a frozen harness (`lemans` ), a minimal agent (`miniswen` ), and one bash tool. Every model saw the same lightweight prompt and the same step/cost limits. Each task was a realistic bug report, security finding, feature request, performance fix, or flaky test on the Writebook codebase. Crucially, the task descriptions never named the Rails API they hinged on - the model had to know it. Judging was by hidden behavior tests: pass/fail on behavior, not implementation style. API recall was scored separately. ## The leaderboard: accuracy and cost | Model | Accuracy | Cost notes | |---|---|---| | Claude Opus 5 | 92% (58/63) | ~132x Luna; ~$120 total | | Claude Fable 5 | ~90% default; ~95% possible | Refused one pen-test-style security task | | Kimi K3 | 90% | Half of Fable’s bill | | GPT-5.6 Sol | 84% | ~$0.52/run; ~36x Luna | | Muse Spark 1.2 | ~76% | ~3x Sol (~$98) | | GPT-5.6 Luna | 73% default; 89% at xhigh | $0.91 default; $2.34 xhigh | | GLM 5.2 | 67% | Mid-pack API recall | | DeepSeek V4 Flash 0731 | 65% | 8% API recall | ## What stands out **A dollar gets you most of the way.** GPT-5.6 Luna solved 73% of the tasks for $0.91. That is a strong default for low-stakes Rails work. **Every point after that gets expensive.** Opus 5 added 19 points over Luna for roughly 132x the cost. The top cluster - Opus, Fable, Kimi K3 - is meaningfully better, but you pay for it. **Reasoning effort scales sharply on Luna.** Moving Luna from default to `high` jumped it to 86% for $1.36; `xhigh` hit 89% for $2.34. That is nearly Fable/Kimi territory for pocket change. **Knowing Rails is what separates the models.** API recall ranged from 8% (DeepSeek V4 Flash) to 35%. Runs that used the correct Rails API solved 92% of the time. Hand-rolled fixes still solved 87%. But when a model saw the API and built custom code anyway, success dropped to 64%. The lesson: model choice matters, but prompt/harness design that surfaces the right API matters too. **Safety refusals are a real failure mode.** Fable 5 likely hit ~95% but refused a task worded like a penetration-test report. If your agent work touches security, this is worth testing before you commit to a model. ## How it compares to other framework benchmarks Rails is not the only project asking whether agents can work inside a real codebase. Several related benchmarks exist, though none are a perfect apples-to-apples match: - **AppSignal’s Rails vs Django vs Laravel benchmark** compared framework-level agent performance across Rails, Django, and Laravel on a set of backend tasks. It is useful for framework choice, but it does not isolate model differences under the same harness. - **Martin Alderson’s 19-framework token benchmark** measured token throughput and cost across many frameworks and providers. It answers “how fast and cheap is the plumbing,” not “does the model know my framework.” - **ABC-Bench** is a broader agent coding benchmark with tasks in multiple languages and frameworks. It tests general agentic coding, not framework-specific API recall. - **7minai constraint decay** work looks at how model performance falls as task constraints pile up - relevant for long-horizon agent runs. - **Reflex full-stack comparison** benchmarks end-to-end full-stack agent workflows, closer to product-building than atomic API fixes. The Rails report is distinctive because it is framework-native: same codebase, same harness, same tasks, with API recall separated from raw accuracy. That makes it a strong complement to general coding leaderboards like SWE-bench. ## Where to use this on Tokenstead We have folded the Rails results into the model cards for every model in the benchmark. Look for the **Agents on Rails benchmark (Aug 2026)** callout on the pages for [Claude Opus 5](/models/claude-opus-5), [Claude Fable 5](/models/claude-fable-5), [GPT-5.6 Sol](/models/gpt-5-6-sol), [GPT-5.6 Luna](/models/gpt-5-6-luna), [Muse Spark 1.2](/models/muse-spark-1-2), [Kimi K3](/models/kimi-k3), [GLM 5.2](/models/glm-5-2), and [DeepSeek V4 Flash 0731](/models/deepseek-v4-flash-0731). If you are choosing an agent harness for Rails work, the [agent harnesses](/agent-harnesses) directory lists the tools that sit between an LLM and your codebase. The benchmark suggests the harness matters almost as much as the model: a harness that surfaces the right Rails API closes a lot of the gap between a cheap model and an expensive one. ## Practical takeaways - **Start with Luna for exploratory work.**$1 for 73% is a remarkable baseline. - **Pay up only when the task is high-stakes.** Opus 5, Fable 5, and Kimi K3 are the safety net for tasks where a missed fix is expensive. - **Tune reasoning effort before swapping models.** Luna’s xhigh run is a reminder that effort settings are a cheap lever. - **Test security tasks specifically.** Fable’s refusal shows that safety alignment can work against you on legitimate security work. - **Build harnesses that expose Rails APIs.** The 92% solve rate when the right API was used is the clearest design signal in the whole report. The benchmark is live at [rubyonrails.org/2026/8/13/agents-on-rails-the-first-benchmark-report](https://rubyonrails.org/2026/8/13/agents-on-rails-the-first-benchmark-report).