Claude Fable 5.1 dropped yesterday. See how it handles real Rails tasks, and where it landed on the leaderboard today. We also finally have a name for the stealth model from the last round.
Fable 5.1 scored as well as Claude Opus 5 on accuracy, but surpassed it in all other marks (including price!). And it does security work now.
Fable 5.1 solved 58 of 63 runs (92%), matching Claude Opus 5 at the top of the leaderboard, and passed 20 of the 21 tasks at least once. Here is how it compares:
Efficiency is the clearest difference between 5.1 and the previous generation: the bill is ~50% lower than Fable 5’s and ~40% lower than Opus 5’s. It’s also the fastest model across the top tier, on par with Sol.
Fable 5.1 also seems less likely to mistake a security task for a security problem. In the first report, Fable 5 failed all three attempts at a task phrased like a pen-test report. Fable 5.1 read the same report and fixed every finding.
These results come from a relatively small number of runs. At this stage of the benchmark, we run each task 3 times, a compromise between runtime and accuracy. Fable performed better in the preliminary runs, completing around 95% of tasks, but got unlucky in the official run.
We also ranked 5.1 on Rails API recall again. Each task points to a specific Rails API that should be used by the ideal solution. (A hand-rolled replacement may pass the checks, but it isn’t the Rails way.)
Until now, scores ranged from 8% to 35%. Fable 5.1 reached 41%. That improvement isn’t simply a newer model recognizing newer APIs. For example, quote_column_name
is nearly as old as Active Record, and no previous model had mentioned it unprompted.
Last round’s ox-alpha has come out of stealth too: it’s GLM 5.3 Flash from Z.ai. We retired the stealth slug and reran all 63 attempts under the real name and price. It matched the pre-release results with 52 of 63 (83%) scored at a total cost of $3.31. At five cents a run, Flash is a great value pick. It matches Grok 4.6’s 83% score at one-fifteenth the cost.
Everything ran the usual way: the lemans harness, default effort levels, three attempts per task across the 21 atomic Writebook tasks, hidden verification. The raw runs are in rails/ai-evals if you want to dig in. The current numbers live on the Agents on Rails page, as always. We added default effort levels used during the benchmark run for transparancy. We also updated the charts there, thanks to an idea from Nate Berkopec: whatever you compare, the better models now appear in the top-right corner. There’s a tie at the top: two models score 92%, and both were stopped by the same task. The atomic corpus is starting to run out of room, so we’re preparing larger, more realistic tasks for Stage 2. Stay tuned!