{"slug": "i-taught-an-open-source-seo-tool-to-check-whether-your-site-is-readable-by-ai", "title": "I Taught an Open-Source SEO Tool to Check Whether Your Site Is Readable by AI Agents", "summary": "A developer added AI-readability checks to the open-source SEO tool open-seo, detecting whether a site blocks AI agents like GPTBot and ClaudeBot or supports the emerging llms.txt standard. The feature distinguishes between training crawlers and live user-request agents, and avoids flagging noise by only reporting when AI agents are treated worse than generic rules.", "body_md": "Most SEO audit tools still ask the questions search engines cared about ten years ago: is your title tag the right length, do your images have alt text, is your sitemap valid. Those still matter. But a growing chunk of how people find content now goes through an AI assistant instead of a search results page, and almost no audit tool checks whether your site is even *reachable* by one.\n\nI picked up an open issue on [open-seo](https://github.com/every-app/open-seo) (an open-source Semrush/Ahrefs alternative) to close that gap, and it turned into a good lesson in scoping a feature for someone else's codebase instead of your own.\n\n\"Is this site AI-readable?\" breaks down into concrete, checkable things:\n\n`robots.txt`\n\nblock the crawlers that build AI search indexes and answer live user requests?`llms.txt`\n\n— the emerging convention that gives an AI assistant a clean map of your important pages?Straightforward to state. The interesting part was not flagging noise.\n\nA site with `Disallow: /`\n\nfor every crawler made a deliberate, site-wide choice. Reporting \"GPTBot is blocked!\" on a site that blocks *everything* isn't useful information — it's noise dressed up as an insight. So the check only flags an AI agent when it's treated **worse than the generic rules**:\n\n``` js\nconst rootUrl = `${origin}/`;\nif ((robots.isAllowed(rootUrl, GENERIC_PROBE_AGENT) ?? true) === false) {\n  return []; // whole site is closed — not an AI-specific signal\n}\n```\n\nI also grouped agents by *why they visit* instead of firing one issue per bot. `GPTBot`\n\nand `ClaudeBot`\n\nare training crawlers; blocking them is often an intentional content policy and barely worth a heads-up. `ChatGPT-User`\n\nand `Claude-User`\n\nfetch a page live because an actual human asked an assistant about it right now — blocking those is a much bigger deal, because you're breaking a request in progress, not opting out of a training set. Same signal, completely different severity, so they're separate issue types with separate default severities instead of one generic \"AI crawler blocked\" bucket.\n\n`llms.txt`\n\nand Markdown alternates are both new enough that almost nobody has them yet. If the check flagged every single audited site for \"missing Markdown alternates,\" it would train users to ignore the audit tool's warnings entirely — the boy-who-cried-wolf failure mode of any linter. Both checks are `info`\n\nseverity with copy that says, explicitly, \"this is common, not an error.\" The one genuinely broken state — a site that *serves* `llms.txt`\n\nbut violates its one hard structural requirement (starts with an H1) — is the only case that gets bumped to `warning`\n\n.\n\nMy first instinct for \"does this page have a Markdown alternate\" was to persist a new boolean column on the crawled-pages table. Then I actually looked at how the project's crawl pipeline works: full page data lives in a database table, but only a *slim summary* (title, status code, a few other fields) survives between crawl phases as durable workflow state — link lists and everything else deliberately get dropped to keep memory bounded on 10,000-page crawls.\n\nAdding a persisted column meant a schema migration, which meant touching both the SQLite and Postgres schema paths this project supports side by side — a much bigger, riskier diff for a feature that doesn't actually need to survive past the audit run. So instead the flag rides through the same transient summary object the pipeline already carries between phases, and the *finding* — not the raw per-page flag — is the only thing that gets persisted, as a single site-level issue row. Same pattern the project already used for the file-based `llms.txt`\n\ncheck; I just extended it instead of inventing a new one.\n\nI'd already opened a PR for the robots.txt and `llms.txt`\n\nchecks. Before anyone had reviewed it, I built the Markdown-alternates check too — and instead of opening a third PR, I pushed a second commit onto the same branch. The maintainer hasn't looked at any of it yet, so there was no in-flight review to disrupt, and one PR that closes three-quarters of the original issue is easier to review than three PRs that each close a quarter of it and reference each other.\n\n19 unit tests, no schema changes, no new external API calls beyond one extra fetch for `llms.txt`\n\n. The PR's open here if you want to see the actual diff: [every-app/open-seo#122](https://github.com/every-app/open-seo/pull/122).\n\nOpen source alternative to Semrush and Ahrefs\n\nOpenSEO is an SEO tool for *the people*. If tools like Semrush or Ahrefs are too expensive or bloated, OpenSEO is a pay-as-you-go alternative that you actually control.\n\nAll-in-one SEO tool for you and your AI agent.\n\nConnect with any agent like Claude Code, OpenClaw or Hermes. We have pre-built skills, but you can build your own to tailor OpenSEO to your needs.\n\nTry OpenSEO for free on our website. If you want to support the project, a hosted subscription is $10/month.\n\nThe code itself was the easy part. The actual work was reading how someone else already solved similar problems in their codebase before writing a single line — the \"how do I avoid noise\" and \"how do I fit this project's existing shape\" questions mattered more than the crawler-detection logic itself.", "url": "https://wpnews.pro/news/i-taught-an-open-source-seo-tool-to-check-whether-your-site-is-readable-by-ai", "canonical_source": "https://dev.to/wahib_el_khadiri_0/i-taught-an-open-source-seo-tool-to-check-whether-your-site-is-readable-by-ai-agents-15i3", "published_at": "2026-07-21 23:08:35+00:00", "updated_at": "2026-07-22 00:02:32.539804+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "ai-agents", "large-language-models"], "entities": ["open-seo", "GPTBot", "ClaudeBot", "ChatGPT-User", "Claude-User"], "alternates": {"html": "https://wpnews.pro/news/i-taught-an-open-source-seo-tool-to-check-whether-your-site-is-readable-by-ai", "markdown": "https://wpnews.pro/news/i-taught-an-open-source-seo-tool-to-check-whether-your-site-is-readable-by-ai.md", "text": "https://wpnews.pro/news/i-taught-an-open-source-seo-tool-to-check-whether-your-site-is-readable-by-ai.txt", "jsonld": "https://wpnews.pro/news/i-taught-an-open-source-seo-tool-to-check-whether-your-site-is-readable-by-ai.jsonld"}}