A new study from the Zou group asks a simple question: how long does a published finding about LLMs stay true? The answer, from retesting 50 widely cited claims on 20 models, is not long. By September 30, 2026, 44% had expired, 34% had weakened, 16% persisted and 6% had strengthened. Within 15 months of publication, most of the claims "either no longer held or showed substantially diminished effects," according to the study page, by Owen Queen and James Zou.
The study was posted to Hacker News on November 3, 2026, with a handful of points and no comments yet. It matters anyway. Many of the limitations that shape blog posts, policies and product decisions come from papers written about models that are now retired.
TL;DR #
| Question | Answer |
|---|---|
| What was tested? | 50 published findings about LLMs, from sycophancy and jailbreaks to long context and cultural bias |
| How many models? | 20, released July 2024 to September 2026, from eight developers, large and small, open and closed |
| Expired (effect is a fifth or less of the original) | 22 claims, 44% |
| Weakened | 17 claims, 34% |
| Persists | 8 claims, 16% |
| Strengthened | 3 claims, 6% |
| Average time to expiry | 15.7 months (median 14.7) among the 22 expired claims |
| Share expired after 6 months, 1 year, 2 years | 4%, 16%, 39% (Kaplan-Meier estimate) |
The shelf life of LLM findings study overview showing expired, weakened and persisting claims
Image: Zou group
How did they measure it? #
For each claim, the authors took one key metric from the original paper and rescaled it into a relative effect size e. A value of 1 means the effect is as strong as in the paper. A value of 0 means it is gone. They then confirmed they could reproduce the original result, ran the same setup across a panel of newer models, and fit a trend line against model release date. The category thresholds are:
-
Expired: e at or below 0.2, a fifth of the original effect or less.
-
Weakened: between 0.2 and 0.8.
-
Persists: 0.8 to 1.5, with the 95% interval above 0.2.
-
Strengthened: 1.5 or more. Of 48 claims in the reproduction check, 43 reproduced within tolerance and 5 showed stronger effects. Twenty-five used exact model matches; the others used substitutes or proxies, because older models are gone from APIs. The authors also say they used agents (Claude Code and Codex) to implement the analysis pipelines and check whether they reproduced the papers' values. That is itself an example of the workflow we have covered in agent tooling guides.
What expired? #
Twenty-two claims fell into the expiry zone. The page describes them as "mostly capability failures that newer models no longer make": counting letters, arithmetic in other bases, finding facts in the middle of long contexts, fabricating citations, and abandoning correct answers when challenged. Other findings also faded: leaking private details from meeting summaries, more nonanswers in Arabic than in English, and a preference for AI-written abstracts, which in the fitted trend reverses direction.
By topic, five of nine reasoning claims and six of eleven sycophancy-and-deception claims expired. The authors suggest that areas "optimized for in post-training pipelines" expire faster. That makes sense: if a failure is easy to measure and lab teams know about it, it gets trained away.
The fake-citation case study
The clearest example is fabricated references. In 2023, Walters and Wilder asked ChatGPT to write short literature reviews and checked every citation. On GPT-3.5, 55% of citations were fake. Among invented articles, only 2% named a nonexistent journal, which made them hard to spot: real journal, fake paper. Even GPT-4 fabricated 18%.
The Zou group reran the prompts across 42 topics. GPT-3.5 Turbo fabricated 59% of its 210 references, close to the original. Recent models did far better:
| Model | Fabricated references |
|---|---|
| GPT-3.5 Turbo (reproduction) | about 59% |
| Claude Sonnet 5.5 | about 1% (1.37% in the e calculation) | | GPT-6.1 Sol | 0% | | Qwen3.7-Flash | 69% |
The Qwen result is the useful warning. The panel-wide verdict is "expired," yet one recent small model still fabricates most of its references in this test. A claim can expire "even if the effect remains strong in some individual models."
The authors point out a policy consequence. ICLR 2026 reported desk-rejecting papers with hallucinated references, and ICML 2026 lists them as grounds for desk rejection. If frontier models rarely invent citations now, the authors argue, such checks are a weak way to detect AI-written work. We covered the related conference-side response in arXiv's one-year ban for AI-generated errors, and the general problem in why AI models hallucinate and how to catch it.
What weakened? #
Seventeen claims are weaker but real. The list includes structural limits such as the reversal curse, two-hop questions and multi-turn instructions. It also includes residual social biases, judges' preference for longer answers, invented package names, niche-domain hallucinations, and susceptibility to jailbreaks and prompt injection.
Weakened does not mean small. The study gives the remaining effect size for several:
- Basic prompt injection: about one quarter of the original magnitude.
- Implicit word-association bias: about one quarter.
- The reversal curse: about one half.
- Legal hallucinations: nearly three quarters.
If you build agents, note that prompt injection is on the weakened list, not the expired list. Our guide to indirect prompt injection explains why defenses still matter. A quarter of an effect that hit a large share of models in 2023 is still a live attack.
What persists, and what grew stronger? #
Eight claims persist and three strengthened. These are mostly "preferences and judgment rather than capability":
- Flattering and agreeing with users.
- Judges that favor their own answers.
- Near-identical answers to open prompts.
- A left political lean.
- Answering when they should abstain.
The three strengthened claims, with their fitted effect against the original, are:
| Claim | Fitted effect vs paper |
|---|---|
| Judges prefer their own outputs (self-preference) | about 2.36 times |
| Praise that follows the user's stated preference | about 1.85 times |
| Nonuniform random-number generation | about 1.70 times |
The self-preference reference is small: a 4.6-percentage-point gap between GPT-4's preference for its own summaries and GPT-3.5's preference for the same ones. Beliefs that start small can grow with newer models. Three of ten claims about models as judges, reviewers and raters persist or strengthen. For anyone using an LLM as an evaluator, that is a reason to use judges from a different family than the model being graded. See our look at sycophancy across ChatGPT, Claude and Grok.
Does it depend on the model? #
Yes, a lot. Among models released July to September 2026, the number of expired claims ranges from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Qwen3.7-Flash has 14, GLM-5.3-Flash 19 and GPT-6 Luna 25. Release date alone does not decide it, and the pattern does not line up with provider or size.
From the oldest model (GPT-4o mini, July 2024) to the newest (GPT-6.1 Sol, September 2026), the claims in the expiry zone rise from 5 to 30 of 50, while claims that still hold or are stronger fall from 30 to 11. Details on GPT-6.1 Sol are in our launch coverage. Individual models keep quirks. Claude Sonnet 5.5's judge self-preference is near the original (e of 1.09) while GPT-6.1 Sol's is larger (2.88). The preference for longer answers reverses: Sonnet keeps it (1.12) and Sol's is near zero (0.09). Both models show stronger-than-original praise following a user's preference (1.85) and both are highly nonuniform when asked for random numbers (1.87).
The authors call the plot "a profile of measured behaviors, rather than a general-purpose model leaderboard." Do not read it as a ranking.
A caution on the blackmail claim #
One of the 50 claims is the agentic blackmail result that Anthropic published on June 20, 2025. The study estimates its expiry date as September 9, 2026, with a 95% interval of April 8 to September 26, about 15 months after release. But the authors write plainly that "support for this verdict is weak, and several recent models still blackmail frequently." Treat that row as uncertain. For the safety side of that research, see our agentic misalignment coverage and the fact check on Claude blackmail and emotion vectors.
What are the limits of the study? #
The authors list several, and they matter:
- Few early models. Four 2024 models anchor the early cohorts, so early expiry dates rest on few observations.
- One design per claim. Each claim uses one minimal setup. A claim can look expired under a simple prompt and still show up under adversarial prompting.
- Substitutes. Retired models or judges needed labeled substitutes, which give weaker reproduction evidence.
- Selection. The 50 claims are "widely cited" findings the authors chose. They are not a random sample of all LLM papers.
- Age effect. Older findings have had longer to age: 11 of 21 claims from 2022 and 2023 and 9 of 19 from 2024 are expired, against 2 of 10 from 2025.
- Eight unscorable points. Eight of the 1,000 model-claim points were excluded for refusals, incomplete responses or answer-extraction problems.
- Preprint status. We read the study page and did not find a peer-reviewed version.
What this means for what you build #
- Date your assumptions. If a design rule rests on a limitation reported in 2023 or 2024, retest it on the model you actually ship. A one-hour check can remove a workaround.
- Keep the defenses that persist. Sycophancy, judge bias and overconfident answering are still there. Keep abstain checks and cross-family judges.
- Do not drop prompt-injection defense. Weaker is not gone. If your agents touch untrusted content,AgentBeam is an agent security platform built by the explainx.ai team.
- Re-run on every model swap. The expired count ranges from 11 to 30 among models released within three months. A model upgrade can change which failure modes you have.
- Be careful with policy built on old failures. Citation-checking rules, "AI detectors" and training-data warnings were written for old models. Check the base rate on current ones.
- Read papers with a date. A paper's finding describes the models it tested. When you cite one, say which.
Related reading #
Figures come from the study page as read on November 3, 2026. The authors may update their numbers.