Downgrading From Claude Fable Barely Affected Dev Productivity During Ban A Harvard University study of 297 Claude Fable 5 users found that access to the higher-scoring model raised the GitHub merge rate by only 1.4 percentage points, despite Fable 5 scoring nearly 11 points higher than Opus 4.8 on the SWE-Bench Pro coding benchmark. The study tracked developers before, during and after the 19-day White House ban on Claude Fable in June, comparing them with 8,908 developers who stayed on Opus 4.8, and found five Claude upgrades produced no significant gain in merged pull requests or weekly completed changes. The results suggest benchmark scores mean little when the tasks developers assign already fall within older models' capabilities. Anderson's Angle https://www.unite.ai/series/andersons-angle/ Downgrading From Claude Fable Barely Affected Dev Productivity During Ban Add Unite.AI to your preferred sources on Google https://www.google.com/preferences/source?q=unite.ai Key PointsAccess to the higher-scoring Claude Fable 5 raised the merge rate by only 1.4… View more - Access to the higher-scoring Claude Fable 5 raised the merge rate by only 1.4 percentage points. - Five Claude upgrades produced no significant gain in merged pull requests or weekly completed changes. - Benchmark scores mean little when the tasks developers assign already fall within older models' capabilities. It is very difficult to accurately assess whether new AI models have improved their performance over their predecessors or not – or even to define what a productivity improvement https://www.unite.ai/productivity-myths-in-software-engineering/ necessarily constitutes, since new releases may demonstrate improvements in one area, but degraded performance in another, in much the same way that conventional software and frameworks can. Sometimes these are voluntary, human-chosen tradeoffs, sometimes ‘collateral damage’. That’s assuming ‘fair play’. There are plenty of cases where developers have rigged software to perform better in benchmarks https://www.gamespot.com/articles/nvidia-accused-of-cheating-in-3dmark-03/1100-6028894/ , sometimes making it recognize when it’s being tested https://www.epa.gov/vw/learn-about-volkswagen-violations and temporarily change how it operates to get a better score. AI models too can recognize when they’re being evaluated https://www.unite.ai/ai-acts-differently-when-it-knows-its-being-tested-research-finds/ , and behave differently, even without being deliberately designed to do so. In search of more meaningful and honest metrics, developers often establish their own casual yardsticks, such as Simon Wilson’s qualitative Pelican riding a bicycle challenge https://github.com/simonw/pelican-bicycle , wherein Vision Language Models VLMs are tasked with creating something that is absent from the training data , and often perform poorly – though things have improved somewhat throughout 2026: However, a benchmark can only measure the particular skills it was designed to test, and even when the results are reliable, it remains unclear how much those skills matter in everyday use, or whether a higher score will translate into more work being completed. In the case of coding models, that can be investigated by looking at what happens to developers’ output when they switch from one model to another. Doing the Same with Less In June this year, the 19-day White House ban https://www.anthropic.com/news/fable-mythos-access on the use of Claude Fable https://www.unite.ai/anthropic-debuts-claude-fable-5-1-and-mythos-5-1-with-split-safeguards/ provided researchers with an unusually clear testing environment, as developers submitting pull requests https://www.ibm.com/think/topics/pull-request PRs to GitHub downgraded to less powerful versions of Claude. Since all versions of Claude co-sign PRs, and since that information is searchable via the GitHub API, the data is freely available. This opportunity was taken up in a new study https://arxiv.org/pdf/2611.23216 from Harvard University, in which the work of 297 Fable 5 users was tracked before, during and after the ban, and compared with that of 8,908 developers who continued using the older Opus 4.8 https://www.unite.ai/what-opus-4-8-changes-for-anyone-running-agents-on-claude/ model. Fable 5 had previously been awarded a score nearly 11 points higher https://assets.claude.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48.pdf page=148 than Opus 4.8 on the SWE-Bench Pro coding benchmark. In the Harvard study, however, access to Fable 5 was associated with an increase of just 1.4 percentage points in the proportion of submitted code changes accepted into GitHub projects within seven days, a difference the paper claims to be practically a rounding error; and a wider analysis also found no significant productivity gains from upgrading to other newer AI models: ‘Results show that having access to Fable 5 changed the merge rate by only 1.4 percentage points, and the change was not statistically significant . We also analyzed five of the eight model upgrades in 2026 and found that the merge rate did not rise after these upgrades either . ‘Similarly, developers did not get more pull requests merged per week. These results show that gains on current coding benchmark did not really translate into more finished work.’ Across five upgrades in 2026, involving 1,208 developers and more than 15,000 proposed code changes, no statistically significant improvement was found in the rate at which submitted code was merged, or in the number of changes successfully completed each week. Though the paper acknowledges that GitHub merges are an imperfect measure of productivity, the ban offered a very rare opportunity to assess the effects of switching models under real working conditions, in the context of a sector that attracts huge global interest, but which operates under the maximum possible corporate opacity. Acknowledging the difficulty of establishing empirical criteria in such cases, the new work