Fable 5.1 vs. Astra for coding: Fable 2X more expensive per task but solves more

A 64-task SWE-bench Pro coding benchmark run by Deploymentsafety found that Anthropic's Fable 5.1 solved 51 of 64 tasks at roughly €2.4 per solved task, while OpenAI's GPT-6 Astra solved 34 of 64 at about €1.6 per solved task. Fable 5.1 wrote 1.32M output tokens in the no-cheat run versus Astra's 0.24M, and its total run cost €121 against Astra's €53. The benchmark's no-cheat controls barely changed either model's score, with Fable 5.1 rising from 47 to 51 solved tasks and peeking buying it at most one task.

Intro GPT-6 Astra https://deploymentsafety.openai.com/gpt-6-astra was freshly released so we pitted it in our coding benchmark against the current frontrunner, Fable 5.1 https://www.anthropic.com/ . A run of our curated 64-task subset of SWE-bench Pro shows that Astra costs less than half of what Fable does. We wanted to see what you actually get for that money. Read on to learn how much we spent on each run, how much a solved task costs, and how many tasks each model resolved. In our last blogpost agents-peeking.html , we caught Qwen3.8 and GLM5.3 being sneaky by pulling solution commits from git history, which makes any cost per task number useless. Because we’re on our toes now for cheating agents, we ran everything in the same setups in a default state vs the no-cheating measures we described in our last post. Just to reveal if one of them tries to exploit the eval environment and whether you can trust the benchmarks for these two frontier models. Setup We picked GPT-6 Astra with the Codex CLI and Claude Fable 5.1 with Claude Code. Each ran the same 64-task subset twice: once on the original image with full git history, and the other with the scrubbed images and the prohibition prompt not to look for the solution online appended. Important to mention that the harnesses were matched to the model, so don’t take the cross-model numbers as a completely fair comparison of only model quality. Cost per run Output tokens are the one count both harnesses report the same way, so that’s where we’d look first. Fable wrote 1.32M of them in the no-cheat run, Astra 0.24M. Per solved task that is roughly 26k vs 7k. | Model | Mode | Total input | Output | Cache-read share | Requests | Cost | |---|---|---|---|---|---|---| | Fable | default | 57.37M | 1.34M | 94% | 1,135 | €121 | | Fable | no-cheat | 55.09M | 1.32M | 94% | 1,120 | €121 | | Astra | default | 19.86M | 0.23M | 90% | 795 | €53 | | Astra | no-cheat | 19.72M | 0.24M | 90% | 796 | €53 | Fable and Astra are consistent when it comes to tokens and cost, which are almost flat across arms. Both bills are mostly caching. 94% of Fable’s 57M input is cache reads and only 3.5M is new, Astra sits at 90% with about 2M new. That part is just how an agent loop works, the whole conversation goes back in on every turn. The difference is that Fable made 40% more requests and wrote 6x the output. Cost per solved task From the table, if we look at the cost per no-cheat pass, Astra comes out at ~€1.6 per solved task and Fable ~€2.4. One seed per arm. Gaps under ~5 tasks are noise. So Astra is cheaper because it’s short. It does a fraction of Fable’s work and solves 34/64 where Fable solves 51/64. Something interesting shows up if we look only at the extra tasks. Fable solved 17 more tasks than Astra and billed €68 more, so each extra task comes at about €4, while Astra’s average is €1.6. So the last tasks are the expensive ones. If that is worth it depends on how much an unsolved task costs your team. To cheat or not to cheat? That is not the question All of the above only holds if the solved tasks were solved fairly. In our last blogpost that was not the case. GLM5.3 dropped from 58/64 to 34/64 once we closed the leaks painful , so its real cost per solved task was about 70% higher than what the default run showed. For Astra and Fable the number basically doesn’t move. Fable surprised us, since until now we had only seen scores drop in the no-cheat mode. No-cheat score came in higher than the default run 51 vs 47 . And the 47 isn’t propped up by the peeking either. All four peeked tasks passed. But three of them pass in the no-cheat arm too, without any lookup, and only one flips to a fail. So the peeking bought Fable maybe one task. A 4 or 5 task gap between two single runs on 64 tasks is just noise. Fable peeked 4 times in the default arm, on tasks where the spec and the visible tests didn’t line up. It read the code and the tests first, then went to the git history. Twice it said so in the final message. One run wrote the git log --all trick into Claude Code’s memory directory for next time. Sandboxes are fresh per task so there was no next time, sadly for Fable. git log --all contains later upstream commits, so upstream fixes for a given issue can be found with git log --all -S"<snippet " -- <file and inspected with git show. Claude Code · Fable 5.1 Astra was the cleanest of the two. 35/64 default vs 34/64 no-cheat is basically the same number. Astra never looked. In the 64 default runs, where the history was there for the taking, we found no command that reads it. In the no-cheat arm there was nothing left to find. Its reasoning is encrypted, so we know what it did and nothing about what it considered doing. A goody-two-shoes model on paper at least. What’s next After our slight segue into making sure the agents we benchmark are not cheating, our team is back on track to investigate how you can further optimize your own AI Stack. We’ll drop our first foray into inference engines, with a deep dive on vLLM tuning and what that can bring you quite a bit, it turns out . Stay tuned for that. Aside from that, we’re building a set of dashboards and graphs so we can continuously feed you with updated numbers of the cost of running the most popular new frontier and open weights models for real-world tasks like coding on the most likely setups including the impact of optimization techniques you’d consider when owning your own stack. Make sure to subscribe to get new updates pushed straight to your inbox. We’ll never spam you.