In 2025 alone, 48 new large language model (LLM) architectures were added to the Hugging Face transformers library, and many of them arrive with a technical report that presents the model as new. What is actually new in these models, and is it tested? We put this question to LLM technical reports. W
The novelty exposed: ablation, reproducibility and transparency in 40 technical reports of open-weight large language models (LLMs)
A study of 40 technical reports for open-weight large language models found that many present models as new without testing what is actually novel, against a backdrop of 48 new LLM architectures added to the Hugging Face transformers library in 2025 alone. The research examined ablation, reproducibility and transparency in those reports, questioning whether claimed novelty is substantiated by experiments.
Run ablations that isolate each claimed architectural change before calling a model new, and document the setup so others can reproduce the comparison.
more in playbook: Evals and testing for LLM appsRun your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.