Originally published on Sitecheckly. The survey found that 6.2% of sites with readable robots.txt files block an AI search crawler, while 15% block a training crawler. Across analyzed home pages, 16% publish llms.txt, and 13% run a detectable end-of-life or vulnerable component. That software figure is a lower bound because detection depends on exposed versions. These findings come from the Sitecheckly open dataset.
The October 2026 Sitecheckly survey used the Tranco top 10,000, list 7462X, covering 2026-10-02 to 2026-10-31. Home pages were fetched on 2026-11-01; 4,934 of 10,000 requested domains could be analyzed. The ranking came from Tranco. These are home-page benchmarks, not whole-site assessments.
Use the website security statistics below to identify checks worth running on your own pages. A common finding is not automatically your most urgent problem. Confirm what applies to your application before changing it. The average security score was 77.1 out of 100. The most common missing header was Cross-Origin-Opener-Policy, but the appropriate response depends on what the page does.
The following shares apply to the analyzed home pages:
| Missing header | Share | Owner action |
|---|---|---|
| Cross-Origin-Opener-Policy | 95% | Assess cross-origin isolation needs before choosing a policy. |
| Referrer-Policy | 73% | Choose a policy that limits unnecessary URL sharing. |
| Content-Security-Policy | 66% | Build and test a policy for the resources the page needs. |
| X-Content-Type-Options | 54% | Set the header to the nosniff value. |
Use the security headers guide for implementation details. A missing header is a configuration finding to assess, not proof of an exploitable vulnerability or a successful attack. Two separate findings need their own review:
Do not put script integrity work in the same task as response-header configuration. Give each change an owner and test it against the page's actual dependencies.
To check your own page, run the free, passive security audit. It checks the URL entered, requires no sign-up, and does not audit every page on the site.
The survey detected at least one end-of-life or known-vulnerable component on 13% of analyzed home pages. Only common components with exposed versions could be checked, so this is a lower-bound detection figure, not a complete software inventory.
Each row below uses its own exposed-version sample. End-of-life and known-vulnerability categories can overlap. Do not add the shares.
| Technology | Exposed-version sample | End of life | Known vulnerability | Action |
|---|---|---|---|---|
| jQuery | 661 sites | 30% | 44% | Upgrade and test dependent scripts. |
| WordPress | 201 sites | 0.0% | 3.5% | Update core and verify compatibility. |
| Font Awesome | 147 sites | 45% | 0.0% | Move to a supported release. |
| Nginx | 146 sites | 70% | 57% | Check vendor support and patch status. |
| Drupal | 135 sites | 7.4% | 0.0% | Update core and modules. |
| PHP | 110 sites | 42% | 42% | Upgrade the runtime and test the application. |
| Bootstrap | 95 sites | 52% | 41% | Upgrade and test layouts and scripts. |
An advisory match needs verification. A Linux distribution may backport a security fix without changing the exposed version string, which can make version-based risk estimates too high. Conversely, a component that exposes no version may escape these checks. The methodology explains these detection limits.
For your own maintenance queue, record the detected version, the installed package's patch status, and the team responsible for the upgrade. Do not treat a hidden version as evidence that software is current. The average AI-search readiness score was 66.4 out of 100. The survey found gaps in structured data and in how pages presented answers.
| Finding | Share |
|---|---|
| No Organization schema | 72% |
| No concise answer-first paragraphs detected | 70% |
| No question-style subheadings | 68% |
| No Schema.org structured data found | 51% |
Treat these as review prompts, not a template every page must follow. Use accurate, relevant structured data. Put a direct answer near the start when the page addresses a question. Use question-style headings when they make the content easier to navigate, not simply to satisfy a check.
The free AEO audit checks the URL entered and can help identify these findings. Passing the checks does not guarantee an AI-search citation.
llms.txt adoption was 16%. That is an adoption measure, not a requirement for visibility. The file remains a proposal, and major engines have not committed to reading it. Google's guidance for AI features in Search says no new AI text files are needed to appear in those features. Keep that distinction in mind when reviewing AI crawler access and related files: publishing an optional file should not displace work on intended crawler access or useful page content.
Among the 4,546 sites with readable robots.txt, 6.2% blocked at least one AI search crawler, compared with 15% blocking at least one training crawler. In that same group, 9.2% blocked training crawlers while allowing AI search crawlers.
These AI crawler blocking statistics show why a blanket “AI bots allowed or blocked” label is not precise enough. Search access and training permissions need separate decisions.
Reported blocking shares for selected agents were:
| User agent | Share blocking it |
|---|---|
| CCBot | 12% |
| Bytespider | 11% |
| GPTBot | 11% |
| ClaudeBot | 10% |
| Google-Extended | 9.3% |
All shares in this table use the readable-robots.txt group, not every analyzed home page.
Check the purpose of each agent before changing policy:
Blocking a search crawler can affect that engine's ability to fetch content. Allowing it does not promise inclusion or citations.
Write down your intended policy, then test representative URLs against the actual robots.txt rules. Under RFC 9309, the longest matching rule wins. Do not assume that the first rule in the file decides the result.
The website SEO statistics point to several practical page-level checks:
| Finding | Share | Owner action |
|---|---|---|
| Images missing alt text | 61% | Write useful alternatives where needed. |
| Title length outside the recommended range | 49% | Review titles for clarity rather than length alone. |
| No h1 found | 34% | Add a descriptive main heading where appropriate. |
| No canonical URL | 31% | Review canonicalization and the intended preferred URL. |
Run the free SEO audit on the URL you want to review. Use the results to investigate the page, not to make automatic edits without context.
The survey also detected these CMS shares:
| CMS | Share of analyzed home pages |
|---|---|
| WordPress | 12% |
| Drupal | 3.6% |
| Adobe Experience Manager | 2.4% |
These are detection shares, not a complete CMS census. Maintain your own platform inventory and name a maintenance owner for each system. That inventory should be more authoritative than what an external home-page check can identify.
There is also a separate front-end dependency finding: 31% of 680 sites exposing a jQuery version ran jQuery 1.x or 2.x. This is a different version-exposing group from the software-health table above.
Use that finding as a prompt to inventory front-end dependencies, plan upgrades, and test compatibility. Keep the two jQuery samples separate when comparing results or reporting progress. The survey analyzed home-page responses fetched over HTTP or HTTPS, without rendering JavaScript. It also included checks beyond those responses, such as DNS CAA records. It respected robots.txt. Domains that could not be analyzed were mostly CDN, DNS and API hosts, sites asking crawlers to stay away, or sites blocking automated requests.
The scope and fetch dates are stated near the top of this report. The methodology describes the collection approach and its limits.
Read the benchmarks with these constraints in mind:
The reported figures and denominators are available in the open dataset, licensed under CC BY 4.0.
What to do next
Does a missing security header mean a site is compromised?
No. A missing header is a configuration finding, not evidence that an attack has succeeded. Its importance depends on the application and the protection the header would provide.
Can a site block training crawlers without blocking AI search crawlers?
Yes. Operators can write separate robots.txt rules for different user agents, then check that the rules reflect their intended policy.
Does an exposed vulnerable version prove the installation is unpatched?
No. A distribution may backport a security fix without changing the version string used for detection. Check the vendor's patch information before assessing the risk.
Can Sitecheckly audit every page in a site at once?
No. Each run audits the single URL entered. The security audit is separate from the SEO and AEO audits.
Want to see how your own site does? Run a free audit at sitecheckly.com. Figures in this article come from Sitecheckly's own audit data; see the methodology.