Most SEO audit tools still ask the questions search engines cared about ten years ago: is your title tag the right length, do your images have alt text, is your sitemap valid. Those still matter. But a growing chunk of how people find content now goes through an AI assistant instead of a search results page, and almost no audit tool checks whether your site is even reachable by one.
I picked up an open issue on open-seo (an open-source Semrush/Ahrefs alternative) to close that gap, and it turned into a good lesson in scoping a feature for someone else's codebase instead of your own.
"Is this site AI-readable?" breaks down into concrete, checkable things:
robots.txt
block the crawlers that build AI search indexes and answer live user requests?llms.txt
— the emerging convention that gives an AI assistant a clean map of your important pages?Straightforward to state. The interesting part was not flagging noise.
A site with Disallow: /
for every crawler made a deliberate, site-wide choice. Reporting "GPTBot is blocked!" on a site that blocks everything isn't useful information — it's noise dressed up as an insight. So the check only flags an AI agent when it's treated worse than the generic rules:
const rootUrl = `${origin}/`;
if ((robots.isAllowed(rootUrl, GENERIC_PROBE_AGENT) ?? true) === false) {
return []; // whole site is closed — not an AI-specific signal
}
I also grouped agents by why they visit instead of firing one issue per bot. GPTBot
and ClaudeBot
are training crawlers; blocking them is often an intentional content policy and barely worth a heads-up. ChatGPT-User
and Claude-User
fetch a page live because an actual human asked an assistant about it right now — blocking those is a much bigger deal, because you're breaking a request in progress, not opting out of a training set. Same signal, completely different severity, so they're separate issue types with separate default severities instead of one generic "AI crawler blocked" bucket.
llms.txt
and Markdown alternates are both new enough that almost nobody has them yet. If the check flagged every single audited site for "missing Markdown alternates," it would train users to ignore the audit tool's warnings entirely — the boy-who-cried-wolf failure mode of any linter. Both checks are info
severity with copy that says, explicitly, "this is common, not an error." The one genuinely broken state — a site that serves llms.txt
but violates its one hard structural requirement (starts with an H1) — is the only case that gets bumped to warning
.
My first instinct for "does this page have a Markdown alternate" was to persist a new boolean column on the crawled-pages table. Then I actually looked at how the project's crawl pipeline works: full page data lives in a database table, but only a slim summary (title, status code, a few other fields) survives between crawl phases as durable workflow state — link lists and everything else deliberately get dropped to keep memory bounded on 10,000-page crawls.
Adding a persisted column meant a schema migration, which meant touching both the SQLite and Postgres schema paths this project supports side by side — a much bigger, riskier diff for a feature that doesn't actually need to survive past the audit run. So instead the flag rides through the same transient summary object the pipeline already carries between phases, and the finding — not the raw per-page flag — is the only thing that gets persisted, as a single site-level issue row. Same pattern the project already used for the file-based llms.txt
check; I just extended it instead of inventing a new one.
I'd already opened a PR for the robots.txt and llms.txt
checks. Before anyone had reviewed it, I built the Markdown-alternates check too — and instead of opening a third PR, I pushed a second commit onto the same branch. The maintainer hasn't looked at any of it yet, so there was no in-flight review to disrupt, and one PR that closes three-quarters of the original issue is easier to review than three PRs that each close a quarter of it and reference each other.
19 unit tests, no schema changes, no new external API calls beyond one extra fetch for llms.txt
. The PR's open here if you want to see the actual diff: every-app/open-seo#122.
Open source alternative to Semrush and Ahrefs
OpenSEO is an SEO tool for the people. If tools like Semrush or Ahrefs are too expensive or bloated, OpenSEO is a pay-as-you-go alternative that you actually control.
All-in-one SEO tool for you and your AI agent.
Connect with any agent like Claude Code, OpenClaw or Hermes. We have pre-built skills, but you can build your own to tailor OpenSEO to your needs.
Try OpenSEO for free on our website. If you want to support the project, a hosted subscription is $10/month.
The code itself was the easy part. The actual work was reading how someone else already solved similar problems in their codebase before writing a single line — the "how do I avoid noise" and "how do I fit this project's existing shape" questions mattered more than the crawler-detection logic itself.