{"slug": "the-first-rails-agent-benchmark-what-it-means-for-picking-a-model", "title": "The first Rails agent benchmark: what it means for picking a model", "summary": "Ruby on Rails published the first public, same-harness benchmark of frontier models performing real Rails tasks, testing 8 models across 21 atomic tasks with 504 runs at a total cost of $491. Claude Opus 5 led with 92% accuracy (58/63) at roughly 132 times the cost of GPT-5.6 Luna, which achieved 73% accuracy for $0.91 per run. The benchmark reveals that API recall varies widely (8% to 35%) and that using the correct Rails API led to 92% task success, while safety refusals can reduce effective accuracy, as seen with Claude Fable 5.", "body_md": "**Ruby on Rails just published the first public, same-harness benchmark of frontier models doing real Rails work.** Eight models, 21 atomic tasks, 504 runs, $491. The result is a clean cost-versus-accuracy curve that is immediately useful for anyone pointing an agent at a Rails codebase.\n\n## What the benchmark actually tested\n\nThe Rails team used a frozen harness (`lemans`\n\n), a minimal agent (`miniswen`\n\n), and one bash tool. Every model saw the same lightweight prompt and the same step/cost limits. Each task was a realistic bug report, security finding, feature request, performance fix, or flaky test on the Writebook codebase. Crucially, the task descriptions never named the Rails API they hinged on - the model had to know it.\n\nJudging was by hidden behavior tests: pass/fail on behavior, not implementation style. API recall was scored separately.\n\n## The leaderboard: accuracy and cost\n\n| Model | Accuracy | Cost notes |\n|---|---|---|\n| Claude Opus 5 | 92% (58/63) | ~132x Luna; ~$120 total |\n| Claude Fable 5 | ~90% default; ~95% possible | Refused one pen-test-style security task |\n| Kimi K3 | 90% | Half of Fable’s bill |\n| GPT-5.6 Sol | 84% | ~$0.52/run; ~36x Luna |\n| Muse Spark 1.2 | ~76% | ~3x Sol (~$98) |\n| GPT-5.6 Luna | 73% default; 89% at xhigh | $0.91 default; $2.34 xhigh |\n| GLM 5.2 | 67% | Mid-pack API recall |\n| DeepSeek V4 Flash 0731 | 65% | 8% API recall |\n\n## What stands out\n\n**A dollar gets you most of the way.** GPT-5.6 Luna solved 73% of the tasks for $0.91. That is a strong default for low-stakes Rails work.\n\n**Every point after that gets expensive.** Opus 5 added 19 points over Luna for roughly 132x the cost. The top cluster - Opus, Fable, Kimi K3 - is meaningfully better, but you pay for it.\n\n**Reasoning effort scales sharply on Luna.** Moving Luna from default to `high`\n\njumped it to 86% for $1.36; `xhigh`\n\nhit 89% for $2.34. That is nearly Fable/Kimi territory for pocket change.\n\n**Knowing Rails is what separates the models.** API recall ranged from 8% (DeepSeek V4 Flash) to 35%. Runs that used the correct Rails API solved 92% of the time. Hand-rolled fixes still solved 87%. But when a model saw the API and built custom code anyway, success dropped to 64%. The lesson: model choice matters, but prompt/harness design that surfaces the right API matters too.\n\n**Safety refusals are a real failure mode.** Fable 5 likely hit ~95% but refused a task worded like a penetration-test report. If your agent work touches security, this is worth testing before you commit to a model.\n\n## How it compares to other framework benchmarks\n\nRails is not the only project asking whether agents can work inside a real codebase. Several related benchmarks exist, though none are a perfect apples-to-apples match:\n\n-\n**AppSignal’s Rails vs Django vs Laravel benchmark** compared framework-level agent performance across Rails, Django, and Laravel on a set of backend tasks. It is useful for framework choice, but it does not isolate model differences under the same harness. -\n**Martin Alderson’s 19-framework token benchmark** measured token throughput and cost across many frameworks and providers. It answers “how fast and cheap is the plumbing,” not “does the model know my framework.” -\n**ABC-Bench** is a broader agent coding benchmark with tasks in multiple languages and frameworks. It tests general agentic coding, not framework-specific API recall. -\n**7minai constraint decay** work looks at how model performance falls as task constraints pile up - relevant for long-horizon agent runs. -\n**Reflex full-stack comparison** benchmarks end-to-end full-stack agent workflows, closer to product-building than atomic API fixes.\n\nThe Rails report is distinctive because it is framework-native: same codebase, same harness, same tasks, with API recall separated from raw accuracy. That makes it a strong complement to general coding leaderboards like SWE-bench.\n\n## Where to use this on Tokenstead\n\nWe have folded the Rails results into the model cards for every model in the benchmark. Look for the **Agents on Rails benchmark (Aug 2026)** callout on the pages for [Claude Opus 5](/models/claude-opus-5), [Claude Fable 5](/models/claude-fable-5), [GPT-5.6 Sol](/models/gpt-5-6-sol), [GPT-5.6 Luna](/models/gpt-5-6-luna), [Muse Spark 1.2](/models/muse-spark-1-2), [Kimi K3](/models/kimi-k3), [GLM 5.2](/models/glm-5-2), and [DeepSeek V4 Flash 0731](/models/deepseek-v4-flash-0731).\n\nIf you are choosing an agent harness for Rails work, the [agent harnesses](/agent-harnesses) directory lists the tools that sit between an LLM and your codebase. The benchmark suggests the harness matters almost as much as the model: a harness that surfaces the right Rails API closes a lot of the gap between a cheap model and an expensive one.\n\n## Practical takeaways\n\n-\n**Start with Luna for exploratory work.**$1 for 73% is a remarkable baseline. -\n**Pay up only when the task is high-stakes.** Opus 5, Fable 5, and Kimi K3 are the safety net for tasks where a missed fix is expensive. -\n**Tune reasoning effort before swapping models.** Luna’s xhigh run is a reminder that effort settings are a cheap lever. -\n**Test security tasks specifically.** Fable’s refusal shows that safety alignment can work against you on legitimate security work. -\n**Build harnesses that expose Rails APIs.** The 92% solve rate when the right API was used is the clearest design signal in the whole report.\n\nThe benchmark is live at [rubyonrails.org/2026/8/13/agents-on-rails-the-first-benchmark-report](https://rubyonrails.org/2026/8/13/agents-on-rails-the-first-benchmark-report).", "url": "https://wpnews.pro/news/the-first-rails-agent-benchmark-what-it-means-for-picking-a-model", "canonical_source": "https://tokenstead.ai/guides/agents-on-rails-first-benchmark-report", "published_at": "2026-08-13 18:34:03+00:00", "updated_at": "2026-08-13 19:19:12.576008+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-tools"], "entities": ["Ruby on Rails", "Claude Opus 5", "Claude Fable 5", "Kimi K3", "GPT-5.6 Sol", "GPT-5.6 Luna", "Muse Spark 1.2", "DeepSeek V4 Flash"], "alternates": {"html": "https://wpnews.pro/news/the-first-rails-agent-benchmark-what-it-means-for-picking-a-model", "markdown": "https://wpnews.pro/news/the-first-rails-agent-benchmark-what-it-means-for-picking-a-model.md", "text": "https://wpnews.pro/news/the-first-rails-agent-benchmark-what-it-means-for-picking-a-model.txt", "jsonld": "https://wpnews.pro/news/the-first-rails-agent-benchmark-what-it-means-for-picking-a-model.jsonld"}}