I Taught an Open-Source SEO Tool to Check Whether Your Site Is Readable by AI Agents

A developer added AI-readability checks to the open-source SEO tool open-seo, detecting whether a site blocks AI agents like GPTBot and ClaudeBot or supports the emerging llms.txt standard. The feature distinguishes between training crawlers and live user-request agents, and avoids flagging noise by only reporting when AI agents are treated worse than generic rules.

Most SEO audit tools still ask the questions search engines cared about ten years ago: is your title tag the right length, do your images have alt text, is your sitemap valid. Those still matter. But a growing chunk of how people find content now goes through an AI assistant instead of a search results page, and almost no audit tool checks whether your site is even reachable by one. I picked up an open issue on open-seo https://github.com/every-app/open-seo an open-source Semrush/Ahrefs alternative to close that gap, and it turned into a good lesson in scoping a feature for someone else's codebase instead of your own. "Is this site AI-readable?" breaks down into concrete, checkable things: robots.txt block the crawlers that build AI search indexes and answer live user requests? llms.txt — the emerging convention that gives an AI assistant a clean map of your important pages?Straightforward to state. The interesting part was not flagging noise. A site with Disallow: / for every crawler made a deliberate, site-wide choice. Reporting "GPTBot is blocked " on a site that blocks everything isn't useful information — it's noise dressed up as an insight. So the check only flags an AI agent when it's treated worse than the generic rules : js const rootUrl = ${origin}/ ; if robots.isAllowed rootUrl, GENERIC PROBE AGENT ?? true === false { return ; // whole site is closed — not an AI-specific signal } I also grouped agents by why they visit instead of firing one issue per bot. GPTBot and ClaudeBot are training crawlers; blocking them is often an intentional content policy and barely worth a heads-up. ChatGPT-User and Claude-User fetch a page live because an actual human asked an assistant about it right now — blocking those is a much bigger deal, because you're breaking a request in progress, not opting out of a training set. Same signal, completely different severity, so they're separate issue types with separate default severities instead of one generic "AI crawler blocked" bucket. llms.txt and Markdown alternates are both new enough that almost nobody has them yet. If the check flagged every single audited site for "missing Markdown alternates," it would train users to ignore the audit tool's warnings entirely — the boy-who-cried-wolf failure mode of any linter. Both checks are info severity with copy that says, explicitly, "this is common, not an error." The one genuinely broken state — a site that serves llms.txt but violates its one hard structural requirement starts with an H1 — is the only case that gets bumped to warning . My first instinct for "does this page have a Markdown alternate" was to persist a new boolean column on the crawled-pages table. Then I actually looked at how the project's crawl pipeline works: full page data lives in a database table, but only a slim summary title, status code, a few other fields survives between crawl phases as durable workflow state — link lists and everything else deliberately get dropped to keep memory bounded on 10,000-page crawls. Adding a persisted column meant a schema migration, which meant touching both the SQLite and Postgres schema paths this project supports side by side — a much bigger, riskier diff for a feature that doesn't actually need to survive past the audit run. So instead the flag rides through the same transient summary object the pipeline already carries between phases, and the finding — not the raw per-page flag — is the only thing that gets persisted, as a single site-level issue row. Same pattern the project already used for the file-based llms.txt check; I just extended it instead of inventing a new one. I'd already opened a PR for the robots.txt and llms.txt checks. Before anyone had reviewed it, I built the Markdown-alternates check too — and instead of opening a third PR, I pushed a second commit onto the same branch. The maintainer hasn't looked at any of it yet, so there was no in-flight review to disrupt, and one PR that closes three-quarters of the original issue is easier to review than three PRs that each close a quarter of it and reference each other. 19 unit tests, no schema changes, no new external API calls beyond one extra fetch for llms.txt . The PR's open here if you want to see the actual diff: every-app/open-seo 122 https://github.com/every-app/open-seo/pull/122 . Open source alternative to Semrush and Ahrefs OpenSEO is an SEO tool for the people . If tools like Semrush or Ahrefs are too expensive or bloated, OpenSEO is a pay-as-you-go alternative that you actually control. All-in-one SEO tool for you and your AI agent. Connect with any agent like Claude Code, OpenClaw or Hermes. We have pre-built skills, but you can build your own to tailor OpenSEO to your needs. Try OpenSEO for free on our website. If you want to support the project, a hosted subscription is $10/month. The code itself was the easy part. The actual work was reading how someone else already solved similar problems in their codebase before writing a single line — the "how do I avoid noise" and "how do I fit this project's existing shape" questions mattered more than the crawler-detection logic itself.