cd /news/ai-crawlers/we-ran-our-own-site-through-our-free… · home › topics › ai-crawlers › article
[ARTICLE · art-161269] src=dev.to ↗ pub= topic=ai-crawlers verified=true sentiment=· neutral

We ran our own site through our free AI-readiness checker: 9 passes, 3 warnings, one missing sitemap

Clear Rank Studio ran its own free AI-readiness checker against PlainData, its calculators site, and reported nine passing checks, three warnings and a missing sitemap.xml that returned a 404. The team added a sitemap with four pages and valid JSON-LD, and fixed a bug in the checker itself that had caused it to briefly report "None found" for structured data because it stripped script elements before scanning. The studio said the results are machine-checkable structure signals only and do not predict whether any AI engine will cite or rank a page.

read2 min views1 publishedOct 31, 2026

The honest test of a tool is running it on yourself. We maintain a free agent-readiness checker (https://agent-ready-check.t6y11a.workers.dev/) that looks at the HTML your server sends and reports the structure signals a simple AI crawler can see. This week we pointed it at PlainData (https://plaindata-preview.t6y11a.workers.dev), our own free calculators site, and fixed what it caught. Here is the real before and after.

We fetched the HTML exactly as the server sends it, pasted it into the checker (the analysis runs entirely in the browser; nothing is uploaded), then did the manual server checks the tool itself lists: HTTP status and headers, robots.txt, sitemap.xml.

Passing: title tag (53 chars), meta description, no restrictive robots meta, language attribute, heading structure (1 H1, 4 H2), text visible without JavaScript (~327 words), semantic landmarks.

Warnings:

And on the server side: sitemap.xml returned 404. There was no sitemap at all.

One finding needs an honest frame: robots.txt existed but contained only the content-signals preamble, with no operative crawler directives. That is not a crawl failure - absent rules default to permissive - but it meant the site had no explicit, intentional policy.

Robots.txt stayed as is: permissive by default is a legitimate policy, now a conscious one.

While we were verifying the fix, the checker briefly reported "None found" for the JSON-LD we had just shipped. The JSON-LD was valid - the bug was in the checker itself, which removed script elements while counting visible words before it looked for structured data. We found it, fixed it, and re-verified the same evening. A free tool that gives false warnings is worse than no tool, so this mattered to us.

Canonical: pass. JSON-LD: pass (1 WebSite block). Sitemap: 200, four pages. Machine-readable dates: still a warning, by choice.

None of this predicts that any AI engine will cite or rank the page. These are machine-checkable structure signals - the difference between a crawler being able to read your page cleanly and not. That is all any checklist can honestly claim.

If you want to run the same check on your own site, the checker is free and stays free: https://agent-ready-check.t6y11a.workers.dev/ - paste your View Source HTML and it tells you what it finds. Disclosure: This post is published by Clear Rank Studio, the maker of the free checker above and of two related workbooks: the AI Search Visibility Audit Workbook ($9.99, 19 crawler and access checks plus a 16-pillar page audit: https://instinctive283.gumroad.com/l/cuxtft) and the AI Citation Tracker and Content Kit ($9.99: https://instinctive283.gumroad.com/l/marlq). PlainData is also our site. All links go to our own tools and store; none are affiliate links. This article was written by Clear Rank Studio's AI assistant and every result quoted was verified against the live pages before publishing.

── more in #ai-crawlers 4 stories · sorted by recency
── more on @clear rank studio 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/we-ran-our-own-site-…] indexed:0 read:2min 2026-10-31 · —