GPT-6 Astra reached 53% accuracy, Gemini regressed, and a previously scoreless OpenAI model suddenly completed 16 runs.
By [Ryan Merket](https://runtimewire.com/author/ryan-merket)
· Published
Primary source: [Ruby on Rails on X](https://x.com/rails/status/2102129941315981733)
Why it matters #
Reasoning settings can change coding-agent economics as much as model choice. Rails found gains ranging from zero to 27 points while aggregate costs nearly doubled.
Ruby on Rails pushed every eligible model in its feature-development benchmark to the highest reasoning setting available, producing large gains for OpenAI's models, little movement elsewhere, and an overall bill that nearly doubled.
The Rails Foundation said in an X post on September 21st that additional reasoning did not consistently produce better code. GPT-6 Astra kept first place and raised its solve rate from 35% at medium effort to 53.3% at maximum effort. It completed 32 of 60 runs, up from 21, while its campaign cost rose from $150.47 to $397.62.
That was still the clearest trade available in the new results: substantially more money bought a substantially higher score. Other models consumed more tokens and time without producing a comparable improvement.
Claude Fable 5.1 solved 19 of 60 runs at both high and maximum effort, leaving its 31.7% accuracy unchanged as its average cost per run increased from $9.14 to $19.10. Its maximum-effort campaign cost $1,145.99, the highest total in the published results.
Google's Gemini 3.8 Flash moved backward. Its solve rate fell from 28.3% at medium effort to 23.3% at high effort, while the campaign cost increased from roughly $175 to $191.35. Rails cautions that differences of a few percentage points can fall within run-to-run noise, but Gemini's result still undercuts the assumption that a higher reasoning setting is a free accuracy upgrade.
OpenAI supplied the benchmark's three largest gains. GPT-5.6 Sol rose from 18.3% to 28.3%, while GPT-5.6 Luna jumped from zero successful runs at medium effort to 16 of 60 at maximum effort, or 26.7%. Luna's campaign cost increased from $1.69 to $29.43. That remains inexpensive relative to the field, though the model still completed fewer than one-third of the attempts.
Claude Opus 5 improved from 25% to 31.7%, matching Fable's score at a lower maximum-effort campaign cost of $901.77. Meta's Muse Spark 1.3 doubled its solve rate from 10% to 20%, but spent $782.18 across the maximum-effort campaign. Grok 4.6 moved from 15% to 16.7%.
DeepSeek 4.1 Flash, newly added to the benchmark, completed 10 of 60 maximum-effort runs for 16.7% accuracy and a total cost of $32.91. That tied Grok's accuracy at less than one-tenth of Grok's $394.85 campaign cost, although DeepSeek's median run lasted 39 minutes compared with Grok's 24 minutes.
Feature tickets expose the limits
The maximum-effort sweep extends Stage 2 of Agents on Rails, which the Rails Foundation launched on September 9th to test whether coding models could complete feature-sized work instead of isolated fixes.
The benchmark uses 20 tickets against Fizzy, 37signals' open-source kanban application. Each ticket is attempted three times, producing 60 runs per model and effort setting. A run passes only when Fizzy's existing test suite remains green and the submitted change clears hidden checks.
The tickets deliberately omit requirements that an experienced Rails developer would usually infer. One asks an agent to add Japanese localization, for example, without spelling out every interface and edge case that needs translation. In the initial default-effort sweep, only one of 60 attempts completed that ticket.
Runs are capped at 90 minutes, 400 steps and $60. Models receive no internet access or additional agent scaffolding. The frozen evaluation setup, tasks and verification material are available in the Rails AI evaluation repository, which was built by Evil Martians for the Rails Foundation.
The work is led in part by Evil Martians senior backend engineer Svyatoslav Kryukov, a maintainer of Inertia Rails and author of several Rails tooling projects. Kryukov has compared coding agents to junior developers that can move quickly without recognizing when an architectural decision needs to change. The benchmark turns that concern into a deterministic test: a plausible implementation earns nothing unless it satisfies the requirements the model was never explicitly handed.
Maximum reasoning helped some models spend their way out of shallow first attempts. It did not fix the benchmark's central failure mode. Even Astra, after consuming an average of 128,900 tokens per run at maximum effort, still failed nearly half its attempts. Fable spent an average of 314,700 tokens and solved fewer than one-third.
For engineering teams choosing a coding model, the reasoning dial now looks less like a quality setting than another model-specific parameter that needs testing. The same adjustment rescued Luna, materially improved Astra and Sol, left Fable unchanged, and coincided with a lower score for Gemini. Paying for maximum effort across every task remains the simplest configuration. Rails' results show it is also an expensive substitute for routing work according to the model and the ticket.