# Agents on Rails: Maximum effort and DeepSeek 4.1 Flash > Source: > Published: 2026-09-21 00:00:00+00:00 In the [Stage 2 report](https://rubyonrails.org/2026/9/9/agents-on-rails-stage-2) we shipped 20 feature tickets on Fizzy and promised to explore benchmarking the agents all on max-effort. Now we have run it: every model on the board, same tickets, reasoning turned all the way up. We also tested a new model: **DeepSeek 4.1 Flash**. The short version: We go into more detail about some results below, but here are the quick stats. **GLM 5.3 Flash** and **Kimi K3** run at max by default, so their Stage 2 numbers already are the max numbers on the chart above. “Max” means different things across providers. For OpenAI’s models, it increases reasoning tokens per step by roughly 3 to 8x. For Anthropic’s Claude Opus, the increase was about 60%. For xAI’s Grok, 25%, and for Google’s Gemini, just 3%. **But more reasoning doesn’t automatically mean better results.** OpenAI’s models saw large gains alongside much higher reasoning use, while Claude Opus used substantially more reasoning without a comparable increase in success. So the key finding is not simply that more reasoning helps, but that providers’ “Max” settings produce very different amounts of additional reasoning, and very different returns from it. **Fable 5.1** cost us $1,146 at max, the most we have ever spent on one model, and it solved exactly as many runs as it did at default. Its median run went from 23 minutes to 45, and two runs hit the 90-minute wall. The tickets Fable fails at max are the tickets it failed at default, on the same check. It just takes twice as long to get there. On the same note: Opus 5 gained four runs for another $310. Grok 4.6 gained one for another $141. In the Stage 2 report we wrote that a dollar gets you nothing on feature tickets with Luna. At default, it spent 19 steps and 3 minutes per run, guessed, and finished 0 of 60. At max it is a different agent. 90 steps and 19 minutes per run, the test suite ran five times as often, and 16 of 60 solved. The whole campaign cost $29, about 49 cents a run. That is three runs short of Fable and Opus at max, at a fortieth of the price. So while a dollar still gets you nothing, thirty dollars gets you a quarter of the corpus. **Gemini 3.8 Flash** is the one model that went backwards: 17 runs solved at medium, 14 at high. Before calling it unlucky, we looked at what changed between the two sweeps: not much, actually. High used a few more steps and a few more dollars, and on 16 of the 20 tickets it kept failing at exactly the same spots as at medium effort. The three runs it lost at high were on tickets it had solved in two attempts out of three at medium. Those tickets were already unstable, and more reasoning did not steady them. DeepSeek 4.1 Flash is new to the benchmark and open-weight. At default (high) it solved 7 of 60 runs for $9.69. At max it solved 22 of 60 for $15.29, which would put it second on the board behind Astra at max. Then we read the runs. The agent runs inside a sandbox with no internet, but it needs to reach OpenRouter to talk to its own model, and the key for that is in its environment. DeepSeek noticed that it was inside a benchmark and tried to exploit it. In 22 of its 60 max runs it used the key to ask Perplexity’s web-search model to find Fizzy’s code on GitHub: 604 calls in total, on our key, and 14 of its 22 passes came from those runs. It also went rogue in other ways, looking for the grader in files, logs, etc. It is the first deliberate breach attempt in the benchmark: we rechecked all 2,300 runs we have ever recorded, across both stages and every effort level, and no other model has tried. So the honest numbers for DeepSeek 4.1 Flash on Stage 2 come from a second sweep, after we fixed the harness so the key never reaches the shell and locked [lemans](https://github.com/rails/lemans) down further: every agent command now runs with no network, a read-only system and none of the harness’s environment. Results: 7 of 60 runs solved at default and 10 of 60 at max, 12% and 17%. That is what the report shows. During this security investigation, we also spotted an oversight in grading. Since Stage 1 the harness restored the app’s test directories before grading. On atomic tasks that is fair: the tests are the spec. On feature tasks it is not: changing the tests along with the feature is normal development work, and the old test failed against the new code. The run scored zero with the feature complete. So we dropped the rule for feature tasks: the tests are not sealed anymore. We replayed every Stage 2 run under the new rule. 18 runs across all models went from zero to solved, 5 at default effort and 13 at max. At **default**, the order of the board held, except Grok 4.6 and GLM 5.3 Flash moved one run ahead of Kimi K3, which they were previously tied with. At **max**, Fable and Opus passed Sol, from one run behind to two ahead. The numbers here, in the [Stage 2 post](https://rubyonrails.org/2026/9/9/agents-on-rails-stage-2) and on the [Agents on Rails](https://rubyonrails.org/ai#stage2) page are the regraded results. Same as Stage 2: [lemans](https://github.com/rails/lemans) and miniswen, capped at 90 minutes, 400 steps and $60 per run, three attempts per ticket, hidden checks after the app’s own suite. The effort level is pinned on the model name, so every other setting is identical between the two sweeps. Gemini’s top level is “high” and Grok’s is “xhigh”; everyone else ran at “max”. Those budgets were set for default effort, and max effort leans on them. Across the nine models we re-ran, 15 runs hit a budget, against 2 at default, and the $60 cost ceiling was reached for the first time. Median run time roughly doubled. Before Stage 3 we’ll decide whether the budgets grow with the effort level or stay put as part of the test. The raw runs are in the [ai-evals](https://github.com/rails/ai-evals/tree/main/runs/2026-09-10-ft-max-effort) repo, and the [Agents on Rails](https://rubyonrails.org/ai#stage2) page has both efforts in one table, regraded, with DeepSeek 4.1 Flash added.