A few weeks ago, I started working on a project called LiveNerf, which attempts to answer a pretty simple question: are AI models actually getting worse over time, or are people just perceiving them that way?
If you've spent much time in AI communities, you've probably seen people complaining about models being silently nerfed. Claude in particular seems to get these accusations pretty frequently. People report that tasks their models previously handled without much trouble suddenly require more prompting, more corrections, or just don't work as well anymore. I've experienced things that made me wonder about this myself, but the problem is that it's incredibly difficult to distinguish an actual regression from normal variation in model behavior. I wanted to see if we could measure it.
What is LiveNerf? #
The idea behind LiveNerf is to continuously benchmark models rather than evaluating them once when they're released. Most benchmarks give us a snapshot of model performance. A new model comes out, everyone compares it against the previous generation, and we get some idea of how much better it is. But what happens after that?
If a provider changes something about how a model is served, or if its performance starts drifting for some other reason, there's not really a standardized way for people outside the company to notice. That's what LiveNerf is intended to address. We repeatedly evaluate models against established benchmarks and compare the results over time. The goal is to identify changes in performance, whether they're improvements, regressions, or just ordinary statistical noise. It's also entirely open source, so people can inspect the methodology and point out problems with it.
So, was Opus 5.5 actually nerfed? #
Based on the evidence we've collected so far, it doesn't look like it.
This is Opus 5.5's performance on the same frozen 78-question panel, measured daily against its first 10 days. Across 2,262 samples as of October 22, the scores fluctuate, but there's no obvious sustained decline. The latest score is 59.0%, compared with a baseline average of 58.4%.
There are some pretty substantial day-to-day swings, which is part of why I wanted to build LiveNerf in the first place. Looking at one bad day could easily make you think something had changed. Looking at the full history tells a different story.
Separately, one of the more interesting findings from our initial testing was that Opus 5 and Opus 5.5 produced nearly indistinguishable results in the comparisons we examined, with approximately 99% agreement.
That doesn't mean they're literally the same model, and it doesn't establish that no changes have occurred behind the scenes. It does, however, suggest that the dramatic degradation some people have been reporting isn't reflected in these particular evaluations.
We've been using benchmarks including SWE-bench and SciCode, which cover software engineering and scientific problem-solving respectively. These give us something more consistent to measure than whether a model feels smarter or dumber on a particular day.
There's an important limitation here, though. Agreement between two model versions isn't the same thing as demonstrating that one version has remained stable over time. The stronger evidence for or against a nerf comes from repeated measurements of the same model.
Our current conclusion is therefore fairly narrow: we haven't found convincing evidence of a regression in the behavior we've measured. That might not be the most exciting outcome, but I think it's a pretty interesting one.
We also found problems with the benchmarks #
Something I wasn't expecting when I started working on LiveNerf was how much time we'd end up spending investigating the evaluations themselves.
During validation, we examined 78 answer keys and found that eight were incorrect, with roughly another 30 cases containing some degree of ambiguity. That's a pretty significant problem when you're trying to detect relatively small performance changes.
Imagine you're measuring a model over several weeks and notice that its score has dropped by a few percentage points. You might reasonably suspect that something changed. But if the benchmark contains incorrect answers, ambiguous grading criteria, or tasks with multiple defensible solutions, you need to account for those problems before drawing conclusions.
Otherwise, you can end up measuring the weaknesses of your evaluation rather than changes in the model. Building a system that repeatedly runs benchmarks is one thing. Building one that can reliably distinguish actual changes in model capability from noise is considerably harder.
Why do so many people think Claude is getting worse? #
I don't think the reports should just be dismissed because our initial results haven't supported them. There are several possible explanations, and they aren't mutually exclusive.
For one, the model itself is only part of what you're interacting with. There's also the system prompt, context management, inference settings, and whatever other infrastructure the provider has built around it. Changes to any of those things could potentially affect the experience without requiring changes to the model's weights. There's also the issue of how people use these systems. If you've been using Claude for months, there's a decent chance that the tasks you're giving it today are more complicated than the ones you were giving it when you started. As you become more comfortable with what a model can do, you naturally start pushing it further.
That could create the impression that the model is getting worse even if its underlying capabilities haven't changed. And of course, it's possible that some regressions are real but don't appear in the benchmarks we're currently using.
A model could become worse at following certain instructions, handling long conversations, or performing a specific kind of task while maintaining essentially identical scores on SWE-bench. This is part of why I don't think any single benchmark is sufficient to settle the question.
Why I think continuous benchmarking matters #
The more I've worked on this, the stranger it seems that continuous evaluation isn't already a standard practice.
We have an enormous amount of infrastructure dedicated to comparing new AI models. There are leaderboards, benchmark suites, independent evaluations, and entire communities devoted to figuring out which model is best. But relatively little of that infrastructure is focused on verifying that a model continues performing the same way after release.
People aren't just asking these models random questions anymore. They're using them to write software, conduct research, operate businesses, and increasingly perform tasks autonomously. If a capability that someone depends on changes significantly, they should ideally have some way of knowing.
I also think there's a broader transparency issue. AI providers have access to much more information about their models than the people using them. That's understandable, but it means users are often left trying to infer changes from their own experiences.
Independent evaluations can't eliminate that information gap, but they can help narrow it. And importantly, they can also establish when the evidence doesn't support a popular claim. I think that's just as valuable as finding a regression.
Where LiveNerf goes from here #
The project is still in its early stages, and there's a lot left to improve. I want to expand the number of models and benchmarks we're tracking, improve how we handle ambiguous evaluation results, and develop better methods for detecting statistically meaningful changes over time.
I'm particularly interested in tracking more specific capabilities rather than relying exclusively on aggregate benchmark scores. A model's overall performance might remain stable while a particular capability changes substantially. If we're only looking at the average, we could miss that entirely.
I'd also like to make it easier for other people to reproduce our findings and contribute their own evaluations. One of the things I've enjoyed most about releasing LiveNerf has been watching people examine the project, question the methodology, and suggest ways to improve it.
Ultimately, I want LiveNerf to be useful regardless of whether it finds that models are improving, degrading, or staying exactly the same.
I didn't build it because I was certain that AI companies were secretly nerfing their models. I built it because I thought the question was worth investigating, and I couldn't find an existing project doing quite what I had in mind.
So far, the results for Opus 5.5 haven't supported the nerf hypothesis. We'll keep measuring.
If you're interested in the project, want to examine the methodology, or think we've missed something, I'd be interested in hearing your thoughts.
**GitHub:** [github.com/ninjahawk/livenerf](https://github.com/ninjahawk/livenerf)