A new independent research preprint examining nearly 200,000 code-generation responses found that five frontier large language models continue to invent package names that do not exist on PyPI or npm.
The models’ overall package-hallucination rates were much closer than those measured in earlier research, ranging from 4.62% to 6.10%. The study also identified package names generated by every model in the test group.
The initial analysis found 127 names shared across Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2. After review by PyPI Security and Socket, researchers determined that 53 of those names, 41 on PyPI and 12 on npm, were still available for registration as of April 2026.
An attacker could publish malware under one of these names and wait for an AI coding tool to recommend it to a developer. This technique, known as slopsquatting, exploits package names that AI models repeatedly invent. Because every model in the study generated the same names, one malicious registration could potentially target users across several model providers.
The findings appear in “The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort,” by independent researcher Aleksandr Churilov. His study has been posted to arXiv but has not been peer-reviewed.
The research identifies potential slopsquatting targets, not an active attack. There is no evidence in the study that any of the 53 names have been registered maliciously or used to compromise developers.
Frontier models narrowed the gap without eliminating hallucinations# #
The study replicates the methodology of research presented at USENIX Security 2025, which tested 16 code-generating LLMs and reported average package-hallucination rates of 5.2% among commercial models and 21.7% among open-source models.
Churilov tested five newer models against the same prompt corpus, generating 199,845 responses between April 22 and April 28, 2026. The prompts included approximately 20,000 programming questions collected from Stack Overflow and approximately 20,000 synthetically generated questions, divided between Python and JavaScript.
The measured rates were:
| Model | Python | JavaScript | Overall | | Claude Haiku 4.5 | 5.49% | 2.76% | 4.62% | | Claude Sonnet 4.6 | 6.63% | 2.62% | 5.41% | | Gemini 2.5 Pro | 6.75% | 3.61% | 5.80% | | DeepSeek V3.2 | 6.69% | 3.78% | 5.89% | | GPT-5.4-mini | 7.27% | 3.14% | 6.10% |
The spread between the lowest and highest rates narrowed from 16.5 percentage points in the earlier model cohort to 1.48 percentage points. However, none of the five models improved upon the earlier study’s lowest individual result of approximately 3.6% for GPT-4 Turbo.
GPT-5.4-mini’s result also carries an important qualification. It was tested at minimal reasoning effort and refused 32.14% of prompts, substantially more than the other models. Its hallucination rate was calculated only from compliant, parseable responses, which may not represent the full prompt distribution.
Python package hallucinations exceeded JavaScript hallucinations across all five models, reversing the earlier study’s finding that JavaScript produced more. The author suggests that newer models may have absorbed npm’s naming conventions more effectively than Python’s mix of import names, distribution names, and naming formats.
Shared package names create targets across model providers# #
The researchers compared the unique package names generated by each model and found an initial intersection of 127 names: 109 associated with PyPI and 18 with npm.
These were not merely similar names. Each candidate appeared in output from all five tested models.
Each shared name gives an attacker a single registration target that could surface in output from models offered by Anthropic, OpenAI, Google, and DeepSeek.
The study proposes two possible explanations for this convergence. Models may learn the same incorrect package references from shared public training material, such as tutorials and documentation. They may also independently extrapolate plausible names from ecosystem conventions, producing names that look correct despite not corresponding to standalone packages.
DeepSeek V3.2 and GPT-5.4-mini had the greatest overall overlap between their hallucinated-name sets, with a Jaccard similarity score of 0.343. The paper says this could indicate common training-data patterns or convergent generation behavior, but the results do not establish a training-data relationship between the providers.
Socket’s review removed six npm false positives# #
Churilov contacted Socket on April 29 after attempts to send the npm findings through the registry’s published security channels were unsuccessful. Socket reviewed all 18 npm candidates against the registry and relevant framework module systems.
That review found that the four highest-volume npm entries were valid Ember.js imports:
@ember/service
@ember/object
@ember/routing
@ember/controller
These modules resolve through ember-source
rather than existing as standalone npm packages. An LLM generating an import such as @ember/service
is producing valid Ember code, even though a direct npm registry lookup for that package name returns a 404.
The review also found that ssh-keys
was an existing npm package and that metro-evaluator
was already represented by a security-hold package.
These six names were removed from the exploitable npm set. The remaining 12 appeared in output from every tested model and had no corresponding package, framework resolver, or existing security hold when reviewed. Public examples from the paper include css-color-stop
, dns-sd
, and dom-ains
.
PyPI Security separately reviewed the 109 Python candidates. Its prohibited-name list and normalization protections prevented registration of 68, leaving 41 available.
Together, the reviews reduced the original 127-name intersection to 53 registrable candidates.
Registry lookups alone can misclassify valid imports# #
Socket’s review identified a limitation in the study’s inherited extraction pipeline: absence from a package registry does not always mean that an import is hallucinated.
Frameworks can provide virtual modules, bundled subpackages, aliases, and other import paths that are valid in source code but have no standalone registry entry. Regex-based extraction followed by a registry existence check can therefore classify correct code as a hallucination.
Churilov incorporated Socket’s findings into version 2 of the paper, documenting framework-provided virtual namespaces as a false-positive class and distinguishing the raw 127-name intersection from the 53 names that remained registrable.
The final figure is still a point-in-time estimate. PyPI and npm protections change, names may subsequently be registered, and the two ecosystems were evaluated differently: PyPI Security applied internal automated defenses, while Socket manually investigated the npm candidates.
The results do not represent every AI coding workflow# #
The study tested one response from each model for each prompt rather than measuring package suggestions inside complete coding-agent workflows. Agents that retrieve current registry data before suggesting or installing dependencies may avoid some of these errors.
The measurements also cover specific model snapshots and settings from one week in April. Hosted models can change without retaining the same public identifier, and the older prompt corpus may have appeared in newer models’ training data.
The study also relied on a regex-based extractor inherited from the earlier research. Socket’s review of the Ember.js imports demonstrates how that method can mistake valid framework modules for nonexistent packages without additional ecosystem-specific validation.
Even with those limitations, the shared names demonstrate that package hallucination remains relevant to the software supply chain. Developers should treat dependencies introduced by AI-generated code as unverified until the package name, publisher, history, and relationship to the intended project have been confirmed.
Lockfiles and version pinning can prevent later dependency drift, but they do not make an unfamiliar package safe when it first enters a project. New AI-suggested dependencies should receive the same review as dependencies proposed by an unknown external contributor.