{"slug": "metr-acceleration-interpretation", "title": "METR Acceleration Interpretation", "summary": "METR reported in a November 5, 2026 assessment that Anthropic is likely improving model capabilities on well-specified tasks at roughly twice the rate it did between March 2024 and October 2025, an estimate METR gives with high uncertainty and says could be closer to 1.3x or 3x the historical rate. The 1.9x figure derives from a linear extrapolation of METR's AI index, which the article argues rests on an unidentified statistical model and may be an artifact rather than a fact about the world. METR itself cautions in the report that \"there is significant risk that acceleration in AECI may not map cleanly to acceleration in real-world impact.", "body_md": "The AI measurement organization METR released a [report](https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/) claiming that AI development has accelerated stating:\n\nAs of October 2026, Anthropic is likely improving the capabilities of their models on well-specified tasks at around twice the rate they did between March 2024 and October 2025. This estimate is given with high uncertainty; we would not be surprised if the true acceleration was closer to 1.3x or 3x the historical rate.\n\nThis is clearly relevant to the discussion of recursive self-improvement: the idea that you can use AI to make your AI better faster. And more broadly relevant to the discussion of AI speed-ups for software engineering in general. And, of course, specifically relevant to the Anthropic IPO coming up. So it’s worthwhile to understand exactly what’s being said here.\n\nWhen you look into the report you find this 1.9x number is coming from improvements on their AI index. They have this figure which linearly extrapolates measurements of their AI index over time and find that there’s a step change over the previous rate of improvement.\n\nNote that you are not talking about acceleration in practical effects: merely going faster than you previously would have expected if, for some reason, you believed that this “linear” rate should continue forever.\nThis is unclear and METR buries their acknowledgement quite far down into the report writing: “We believe there is significant risk that acceleration in AECI may not map cleanly to acceleration in real-world impact.”<sup>[1](#fn:1)</sup>.\n\nYou might then, reasonably wonder: what is this AI index? And, perhaps, if you think a little further: why should I interpret this linearly and universally?\n\nYou should not interpret this linearly nor universally. There’s a couple things going on here:\n\n- The AI index is a statement about log odds which are slightly counter-intuitive\n- There’s unidentifiable assumption: there’s many plausible curves that could be fit and crucially we don’t have any data which would allow us to distinguish between them.\n- Even thinking about this as a single dimension seems misleading: there seem to be at least two stories here (math vs non-math)\n\nWhat METR has uncovered may simply be an artifact<sup>[2](#fn:2)</sup> of the statistical model they have chosen: it is not *necessarily* a fact about the world.\n\nLet’s dig in.\n\n## A brief Item Reponse Theory primer[#](#a-brief-item-reponse-theory-primer)\n\nThe METR/Anthropic AI index (AECI), according to the documentation, is constructed in the same way as the Epoch ECI (just with proprietary eval sets).\nThe Epoch AI Index, according to its [documentation](https://epoch.ai/data/eci-documentation/methodology), is a 2-parameter Rasch Item Response model.\nItem Response Theory, in my opinion, is the least objectionable way to conduct evals/benchmarking so let’s unpack this further.\n\nThe general idea is that you have a bunch of items (indexed by $i$ and a bunch of agents (indexed by $j$) to whom you poses the tasks and success/failure is measured by $Y_{i,j}$ (0 if failure and 1 if success). A 2-paramter Rasch model just posits that \n$$\nY_{i,j} \\sim logistic(a_{i}(\\alpha_{j} - \\delta_{i}))\n$$\n\nwhere $\\alpha_{j}$ represents the ability of agent $j$, $\\delta_{i}$ represents the difficulty of item i, and $a_{i}$ represents the discrimination: how sharply the test distinguishes between ability levels.\n\nThis makes sense at the item level but they don’t want to work at the item level and instead the AI index data is aggregated at the benchmark level.\nThis is a little inelegant, as we’ll see later, but we can still proceed.\nThus what we observe is the fraction of tasks passed per benchmark (more-or-less<sup>[3](#fn:3)</sup>).\n\n$$\n\\mu = E[\\frac{1}{n} Y_{i,j}] = logistic(a_{i}(\\alpha_{j} - \\delta_{i}))\n$$\n\nand then assuming a Beta distribution with standard deviation\n\n$$\nSD[\\frac{1}{n} Y_{i,j}] = 2\\sigma_{b}\\sqrt{\\mu(1-\\mu)}\n$$\n\nwhich gets a per-benchmark noise term $\\sigma_{b}$.\n\nYou can fit this sort of model using a nice Bayesian model which I won’t go into as it’s not relevant to this discussion.\nThey do a couple other things that I’m not sure about about<sup>[4](#fn:4)</sup> so for clarity I’ll just use their code to demonstrate everything below.\n\n### Can we replicate acceleration from the public data?[#](#can-we-replicate-acceleration-from-the-public-data)\n\nIndeed the first check is whether we can replicate their finding. We don’t have the proprietary data that METR based their analysis on but there is publically available benchmark results from Epoch. It won’t have the internal models nor the internal benchmarks but you could see a little slope increase in the public models on the private benchmark figure: we should should be able to see something.\n\n```\nfull-panel fit: 254 models, 57 benchmarks\n\nALL benchmarks       n=26 (pre=13, post=13)  ratio=1.516\n\nold-vintage benchmarks: 15, new-vintage benchmarks: 42\n\n  (187 models, 15 benchmarks, all orgs)\nOLD-vintage only     n=23 (pre=12, post=11)  ratio=0.670\n  (192 models, 42 benchmarks, all orgs)\nNEW-vintage only     n=25 (pre=12, post=13)  ratio=1.518\n```\n\nAnd indeed we due; using the same pre/post acceleration logic as the METR report (using their code) we get an estimate of 1.5x. So we’ll proceed assuming similar things hold in the proprietary data.\n\n## The Model Assumptions Strike Back[#](#the-model-assumptions-strike-back)\n\nWe’ve had to make quite strong assumptions with this model, namely:\n\n1. We have a linear relationship between scores: an increase of $\\Delta$ means roughly the same across all evals and for all models. So only the distance matters: not where you are on the particular difficulty curve. This by its nature precludes the potential for diminishing returns.\n2. We have a logistic relationship on the difference of scores: there’s really no reason to assume that this might be true or not. Indeed this additionally problematic because we’re not assessing on items but rather on benchmarks which are collections of items.\n\nLet’s examine each in turn:\n\n### Saturation[#](#saturation)\n\nA linear score relationship implies strange things about the probability of success since we’re pushing this linear change through a logistic function. When scores (and consequently probability of success) are initially low there are large gains to linear increases. Indeed this gets faster and faster until at ~50% success probability it suddenly flips and now you’ve reached diminishing returns: each linear increase in ability yields a smaller increase in probability of success.\n\nNow turning this around: for a constrant rate of probability of succes you see the opposite: it’s slow around 50% success and accelerates towards the tails. Suppose you are above 50% pass rates: you will see a decelleration in probability of success even while seeing acceleration of ability scores. As mentioned in the intro: the gain between 90% probability of success and 99% probability of success: that’s the exact same linear ability improvement as going from 99% to X.\n\nAnd there’s reason to believe we are close to this case:\n\nAnthropic’s internal benchmarks are increasingly saturated: Model 4 achieves greater than 50% on all the AECI benchmarks on which it was tested. However, measured acceleration appears reasonably robust to saturation concerns.\n\nThey measure saturation as >85% which is about where we start to see these effects really bite (in between 0.15 and 0.85 the logistic is reasonably approximated by a straight line as you can see in the earlier figure) Indeed when we look at the public data we see a similar trend occuring\n\nThe METR report tries to patch up this by dropping benchmarks which are saturated. However this is not a sufficient fix: we don’t need a single benchmark to be saturated. We can get this if the mix of benchmarks shifts over time. If harder evals are being phased in then we’ll see this break in the ability-scale show up.\n\nNow let’s see the mix shift effect: we’ll introduce progressively harder tests as we go onwards.\n\nThis matches what we see in practice. Benchmarks get retired once a frontier model can solve them and they get replaced by harder ones: thus ARC_AGI becomes ARC-AGI-2 and so-on).\n\nThis leads to the problem of drift.\n\nMix-shift alone is one half of the story. The other: every benchmark above is calibrated EXACTLY to the frontier at the instant it’s introduced – a designer with perfect, instant knowledge of current ability. Real benchmark design takes time, and more specifically, a lab’s INTERNAL frontier is ahead of what’s publicly known (nobody outside has even heard of the next model yet) – so a benchmark calibrated against the best PUBLICLY known model is calibrated against a stale reference by construction, not by designer error.\n\n```\nreal (margin, discrimination) pairs from 44 benchmarks\nmargin (generation units): median=6.01\ndiscrim (synthetic scale): median=0.0370\n\nFully real-calibrated: real quarterly cadence, margin/discrimination resampled\nfrom the real estimated pairs above -- no assumed lag, no assumed spread. True\nability grows at the real historical constant rate throughout.\n\nseed=0  ratio=1.029  frac_saturated=0.0076\nseed=1  ratio=1.034  frac_saturated=0.0049\nseed=2  ratio=1.052  frac_saturated=0.0037\nseed=3  ratio=1.129  frac_saturated=0.0039\nseed=4  ratio=1.064  frac_saturated=0.0012\n```\n\nSo: can pushing this mechanism harder reach METR’s reported 1.9x [95% CI: 1.5, 2.8]? The obvious lever is to make post-breakpoint calibration worse than the data actually show.\n\n```\nextra_shift = additional generations subtracted from the REAL post-breakpoint\nmargin pool (already era-specific, median -0.2 gen before any shift) -- how much\nWORSE than reality would calibration need to be to approach 1.9x?\n\nextra_shift=  0  mean_ratio=1.019  range=[0.992, 1.065]\nextra_shift= 10  mean_ratio=1.005  range=[0.959, 1.062]\nextra_shift= 20  mean_ratio=0.995  range=[0.942, 1.048]\nextra_shift= 40  mean_ratio=0.985  range=[0.926, 1.027]\n```\n\n## Link Functions[#](#link-functions)\n\nBut there’s another issue: it’s not clear that the logistic choice is correct:\n\nThere’s some weirdness here: these link functions make sense on a per-item basis. But, remember, the ECI is not measured on an item-level: it’s measured on a benchmark level. You basically get a score from the benchmark and assert that collectively it follows this distribution. This is problematic because we have very little reason to believe that benchmarks should exhibit this behavior in aggregate. Indeed you can get arbitrarily weird looking curves by changing up the distribution of difficulties.\n\nFor instance here’s a couple curves I generated just using different distributions of individually logistic items:\n\nEvery item here is exactly logistic; only the spread and shape of item difficulties within the benchmark differ. The tight distribution nearly reproduces a straight line in logit space (all items agree on where the benchmark “is”), the wide one flattens out, and the trimodal one visibly kinks at the ends: none of which you could tell apart without looking into the scores themselves.\n\nWhich, of course, they do and find that “Most of the benchmarks used for Model 4 appear reasonably linear in log-odds space.” These seem to indicate that the logistic link function works? Let’s check them out?\n\n```\nbenchmarks tested: 56\nsignificant curvature (p<0.05): 15 (26.8%)\n            benchmark   n       quad_p\n         GPQA diamond 174 1.309414e-14\n              ARC-AGI  89 1.938476e-08\n           ProofBench  70 2.654082e-07\n        Chess Puzzles  85 1.028010e-05\n                 MMLU  92 1.070647e-05\n         FrontierCode  42 6.473409e-05\n              Cybench  19 7.982069e-05\n          APEX-Agents  55 1.219482e-03\n          SimpleBench  78 2.488593e-03\nSurface Evolver Bench  24 5.952378e-03\n```\n\nLike they do “sort of look linear” but when you run a statistical test on adding a quadratic term you find that 15/56 fail to validate the logistic curve. We don’t have access to the Anthropic sets: you can certainly design benchmarks to follow these and presumably there’s someone smart at Anthropic that could have done so. But hard to say.\n\nThis is a place where I’d admit that all models are wrong: what matters is how bad this affects our decision. Noteably they acknowledge that the link function matters and their estimates for acceleration range from 1.1 to 2.9x depending on the link function. Which does leads to some significant uncertainty but also! remember that there are infinite potential link functions. We can only distinguish between link functions in the regions where we have data. We are particularly blind in the sparse regions where we have little data. Thus we would be more concerned if these violations from our fits tended to be in the high capability range (and especially if they tend to be overestimates of performance).\n\nThus how much does the curvature affect our estimates for the highest-capability models?\n\n```\nsignificantly-curved-benchmark rows: 1087\nhigh-capability subset (capability >= 156.1): n = 267\n\nmedian |residual|, logit space: 0.133\neffective ability offset (local slope): median 1.06 , 90th pct 6.18 , max 34.28\neffective ability offset (global/average slope): median 0.93 , max 6.47\nlocal/global ratio: median 1.15 x, 90th pct 2.33 x, max 5.3 x\n\n|offset| > 2 ECI points: 70 / 267 (26.2%)\n|offset| > 5 ECI points: 35 / 267 (13.1%)\n|offset| > 10 ECI points: 17 / 267 (6.4%)\n|offset| > 20 ECI points: 6 / 267 (2.2%)\n\nbenchmarks contributing offsets > 5 points: ['APEX-Agents', 'ARC-AGI', 'Chess Puzzles', 'GPQA diamond', 'SimpleBench']\n```\n\nFor reference, the whole Claude 3.5 Sonnet -> GPT-5 anchor span is 20 ECI points. Thus we see there is a particularly heavy tail\n\n## Does a single dimension even make sense?[#](#does-a-single-dimension-even-make-sense)\n\nWe’ve been laborouring under this assumption that there is indeed a single “ability score”. This should remind folks of IQ and the infamous g-factor. Like if we applied this reasoning to humans we would catch that this doesn’t make sense: just because someone performs well on a math test does not imply that they would perform well on a medical knowledge test: indeed it might even be the opposite. That said, there still is some latent factor in there that explains both. The question then becomes how explanatory this factor is and crurically what if there’s actually a couple?\n\nSo let’s take a look at a dimensionality analysis of these benchmarks\n\nThis hits particuarly hard when we consider that reinforcement learning from verifiable rewards (RLVR) is in play: like we totally expect math to benefit from this while medicine will not.\n\nTo check let’s look at our model residuals: if there’s a single-factor then those residuals should be iid noise. If however, those residuals still contain structure we know there’s something else going on. The approach: fit the same single-factor (2PL) model used throughout this post, take the logit-space residual of every (model, benchmark) observation from that single-factor fit, and ask whether those residuals still contain structure.\n\n```\nfraction of the model x benchmark grid actually observed: 0.192\n\ncomponent  observed  null_mean  null_p95  p_value\n1          17.56     14.30      14.88     0.0\n2          14.34     13.54      13.93     0.002\n3          12.95     12.73      13.17     0.228\n4          12.52     11.94      12.38     0.018\n5          11.82     11.34      11.75     0.026\n6          11.35     10.81      11.17     0.008\n7          10.94     10.40      10.67     0.002\n8          10.66     10.10      10.33     0.0\n```\n\nThe top residual component sits well above anything the null distribution: there’s a real factor left over after the general ability factor. So what is it?\n\n```\nmost negative loadings on residual factor 1:\n  OTIS Mock AIME 2024-2025            -0.387\n  FrontierMath-Tiers-1-3-v2-Private   -0.292\n  FrontierMath-Tier-4-v2-Private      -0.182\n  MATH level 5                        -0.168\n  Furniture Assembly                  -0.118\n  ProofBench                          -0.112\n  WeirdML                             -0.076\n  VPCT                                -0.043\n  EBR-bench                           -0.025\n  GeoBench                            -0.024\nmost positive loadings on residual factor 1:\n  APEX-Agents                         0.086\n  Aider polyglot                      0.087\n  SimpleBench                         0.143\n  ARC-AGI-2                           0.177\n  Mystery Game Puzzles                0.182\n  GPQA diamond                        0.203\n  SimpleQA Verified                   0.244\n  ARC-AGI                             0.244\n  LMCA                                0.407\n  DTBench                             0.472\n```\n\nIt looks like it’s everything other than math! The negative loading benchmarks are almost all math related and the positive loading benchmarks are all general knowedge / reasoning / coding related.\n\nNote that the sign is interesting here: these are negative loadings.\nIf something were to cause agents to become better at math (**cough** RLVR) then that would show up in the original loading: these additional structure is just needed to capture the residuals better.\n\n```\nmath-loading (< -0.05): 7 benchmarks\nrest-loading (> +0.05): 12 benchmarks\n\n  (163 models, 7 benchmarks)\nMATH-loading n=25 (pre=12, post=13)  ratio=1.341\n  (188 models, 12 benchmarks)\nREST-loading n=25 (pre=12, post=13)  ratio=1.128\n```\n\n## Where does this leave us?[#](#where-does-this-leave-us)\n\nSo we’ve replicated a version of METR’s acceleration claim. Do we believe it?\n\nAt the very least we can conclude that Anthropic’s latest models are likely<sup>[5](#fn:5)</sup> better than their earlier models.\n\nAside from that it’s unclear how one would measure acceleration. As we’ve seen acceleration on ability scores presumes an interval scale for capability and no such scale is inherently implied by bechmark pass-rates. This simply isn’t something you can measure: it’s not that we need more data, any data of this sort just cannot answer this question.\n\nRegardless I think you can find issue with model based scores altogether: they lack interpretability. What would be more interesting is something like comparing to a fixed distribution of ailitities: say get a bunch of humans and measure their ability and then compare against that. This will be tell us whether the gain is moving through a range of relevant task-performance or whether we’re just mesauring artifacts.\n\nThere’s reasons to suspect that there are mechanisms which suggest discounting these estimates: quite hard to say.\n\nThe most damning analysis is this assumption of uniformity. Like it’s quite clear that math and other verifiable fields are being targetted. METR somewhat nods to this with their “well-specified tasks” language but even then they’re conflating between these math benchmarks and general knowledge benchmarks.\n\nBut also, there seems like there’s something there.\n\nPerhaps it’s clear from the redacted material\n\nPersonally I’m Like I certainly used Claude when I was writing up this analysis.\nIt probably help me write it more quickly than I would otherwise.\nIs that generalizable: not really but it does seem to mesh with the researchers interviewed.\nCertainly anecdata and you have to discount due to the koolaid factor but it does seem that folks generally believe that using agentic coding is accelerating their work<sup>[6](#fn:6)</sup>.\nI find it fascinating that they don’t believe that additional resources isn’t a large factor: like it’s quite plausible that they’re research insight constrained but surely the rest helped more than 1.7x?\nBecause I think that’s what this has to be saying; it seems insane that Claude is providing any significant lift on ideas more like reducing the burden of coding up experiments and prototypes.\nBut even supposing that Claude brought this to zero that implied that researchers were spending $X of their bandwidth on that sort of thing?\nAnd it makes such a big deal: like everyone has been emphasizing the bitter lesson and scaling laws and it’s wild that this isn’t considered more seriously.\nThis seems more consistent with the intrepreation that in the last year RLVR started becoming a big deal and METR’s benchmarks are driven by areas that are amenable to this.\nThat’s a much weaker claim than recursive self-improvement as improving LLMs in generality  does not seem to allow easy verification.\nBut does speak to the power of new approaches [even if they don’t see benefits across the board]\n\n1. \nOf course they also hedge the other way:\" If real-world impact is a convex function of benchmark scores, e.g. if impact depends on crossing a capability threshold, then AECI would understate real-world acceleration. In some domains we have seen a steady increase in benchmark scores accompanied by sudden accelerations in real-world impacts, e.g. impacts on the rate of discoveries in cybersecurity or math\" [↩︎](#fnref:1)\n2. \nIndeed they mention as such but then proceed to ignore these caveats The 1.9x acceleration [95% confidence interval: 1.5–2.8x] in the AECI trend since October 2025 described below significantly informed this view, but could be explained by overfitting to the benchmark distribution or otherwise be an artifact of the AECI methodology. [↩︎](#fnref:2)\n3. \nThey recognize that there are likely broken tasks and some ability to randomly guess and they adjust the results accordingly: basically scaling [random_chance, 1 - wrong_fraction] to [0,1]. They then model the mean of these aggregates similarly to the Rasch model [↩︎](#fnref:3)\n4. \nI replicated what I thought was the model but got different results [↩︎](#fnref:4)\n5. \nAside from questions like “what if they’re overfitting to the benchmark” or (taking overfitting to an extreme) “the AI just stole the answers”, c.f. the [OpenAI rogue agent breach of Hugging Face](https://en.wikipedia.org/wiki/OpenAI_rogue_agent_breach_of_Hugging_Face)[↩︎](#fnref:5)\n6. \nKeep in mind METR’s previous report showing that folks who believed AI was saving time were actually slower. [↩︎](#fnref:6)", "url": "https://wpnews.pro/news/metr-acceleration-interpretation", "canonical_source": "https://minimallysufficient.com/posts/metr-irt-interpretation/", "published_at": "2026-11-07 08:00:00+00:00", "updated_at": "2026-11-16 05:19:17.807118+00:00", "lang": "en", "topics": ["ai-research", "ai-safety", "large-language-models", "artificial-intelligence"], "entities": ["METR", "Anthropic", "Epoch AI", "AECI", "Epoch ECI"], "also_reported_by": [], "takeaway": "METR's 1.9x acceleration estimate comes from linearly extrapolating its Rasch-model AI index, and METR warns the true rate could be 1.3x or 3x, with acceleration in AECI possibly not mapping cleanly to real-world impact.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/metr-acceleration-interpretation", "markdown": "https://wpnews.pro/news/metr-acceleration-interpretation.md", "text": "https://wpnews.pro/news/metr-acceleration-interpretation.txt", "jsonld": "https://wpnews.pro/news/metr-acceleration-interpretation.jsonld"}}