A study comparing six token-saving approaches for GPT-5.6 agents across ten real tasks found that none of the methods consistently reduced total tokens in both runs, with only the cheaper 5.6 Terra xh…
An evaluation of Anthropic's Sonnet 5 and Opus 4.8 models across 24 coding tasks shows Sonnet produces clearer and more intentional patches, while Opus yields simpler, more robust, and more minimal di…
GLM 5.2 ranked last in code quality and equivalence among AI models tested on 50 real pull requests from Go and Rust repositories, while costing roughly twice as much as Composer 2.5. The model produc…
Composer 2.5 is 6.5x cheaper than premium models on Rust and Go PRs, but ranks last in code quality. Opus 4.8 leads in craft and local score, making it the best choice for complex tasks, while Compose…
Fable 5 scored 81.7 on 30 real tasks from two open-source repos, outperforming Opus 4.8's 72.7, but the lead is not statistically significant at this sample size. Fable costs $5.29 per task, 1.49x mor…
Opus 4.8 outperformed GPT-5.5, Opus 4.7, and Composer 2.5 on 50 real pull request tasks from the graphql-go-tools and sqlparser-rs repositories, achieving the highest craft score in both Go and Rust. …
A developer used Codex to iteratively improve an AGENTS.md file across eight versions, measuring each against real pull requests on a five-task training set. The best candidate showed improvements in …
A regression check comparing GPT-5.5 high against a prior run on 21 tasks from the GraphQL-go-tools repository found the model resolved one fewer test, produced two fewer equivalent patches, and recei…
An evaluation of Anthropic's Opus 4.7 model across five reasoning effort settings on 29 real-world tasks from an open source GraphQL repository found that medium reasoning effort produced the best res…
OpenAI's GPT-5.5 Codex model was tested at four reasoning effort levels — low, medium, high, and xhigh — on 26 real-world coding tasks from the open-source GraphQL-go-tools repository. Low and medium …
OpenAI's GPT-5.5 outperformed Anthropic's Opus 4.7 and OpenAI's GPT-5.4 on 56 real coding tasks from two open-source repositories, passing more tests and producing patches that more often survived cod…