{"slug": "rankprobe-an-seo-and-ai-crawler-audit-that-fails-your-pr-before-a-stray-noindex", "title": "rankprobe: an SEO and AI-crawler audit that fails your PR before a stray noindex ships", "summary": "A developer released rankprobe, an MIT-licensed Python tool that audits a site for 36 SEO and AI-crawler checks and can fail a pull request in CI when issues like a stray noindex, an overlong meta description, or a robots.txt blocking GPTBot and ClaudeBot are found. Run against the developer's own 15-page Jekyll blog, the tool flagged all 15 pages for meta descriptions over 160 characters and titles over 65 characters, all 15 missing an og:image, and the origin missing robots.txt, sitemap and llms.txt. The tool ships with a robots.txt matcher that reports which AI crawlers are allowed or blocked, plus SARIF, JSON and --fail-under options for CI integration.", "body_md": "I cross-post my dev.to articles to a small Jekyll site on GitHub Pages. Last week I pointed a new audit tool at that site and it told me 15 out of 15 pages had a meta description Google will cut off, 15 out of 15 titles were too long, no page had an `og:image`, and the origin served no `robots.txt`, no sitemap and no `llms.txt`. None of that was visible from the browser. Every page looked fine.\n\nThat tool is **rankprobe**, and this post shows what it does on a real site and how to wire it into CI.\n\n**Repo:** [github.com/bayuh85/rankprobe](https://github.com/bayuh85/rankprobe) (MIT, Python 3.10+, standard library only, 80 offline tests)\n\nSEO breakage is silent. A `noindex` left over from staging, a canonical pointing at the wrong host, a robots.txt that blocks GPTBot and ClaudeBot because someone copied a \"block AI\" snippet, JSON-LD with a trailing comma. The page renders, the PR gets approved, and you find out weeks later from Search Console, or from the fact that no AI search engine ever cites you.\n\nThe usual options are a paid SEO suite (slow, per-seat, not in your CI) or handing the HTML to a chat model and asking \"what's wrong with my SEO?\", which spends tokens rediscovering things a parser settles in milliseconds, and sometimes invents issues.\n\nI wanted the boring part to be deterministic and the model part to be optional and small.\n\n`rankprobe audit <url-or-file>` fetches the page (or reads a local HTML file) and runs 36 checks: status and redirect chain, meta robots and `X-Robots-Tag`, canonical, hreflang, title and description length, one `h1`, heading order, missing `alt`, `lang`, viewport, page weight, Open Graph and Twitter tags, JSON-LD syntax and required fields, plus `robots.txt`, sitemap and `llms.txt` at the origin. Every finding carries the evidence it found.`rankprobe robots <url>` prints which AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, Applebot-Extended) robots.txt allows or blocks. It uses its own matcher: longest match wins, `Allow` wins ties, `*` and `$` work.`rankprobe plan <url>` runs the audit and sends `--max-pages N` crawls same-origin links and flags duplicate titles and descriptions. `--fail-under N` exits 1 for CI. `--sarif` writes SARIF 2.1.0 for GitHub code scanning. `--json` is there for scripts.\n\n```\npip install git+https://github.com/bayuh85/rankprobe\nrankprobe audit https://your-site.dev\n```\n\nHere is the crawl I ran on the blog, 15 pages:\n\n```\nrankprobe audit https://bayuh85.github.io/atmorouter-blog/ --max-pages 15\n```\n\nGrouped by check id, the findings were:\n\n| Severity | Check | Pages | \n|---|---|---|\n| warn | `meta-description` (over 160 chars) | 15 | \n| warn | `title-length` (over 65 chars) | 15 | \n| warn | `og-image` missing | 15 | \n| warn | `canonical-target` (canonical on another origin) | 9 | \n| warn | `duplicate-description` | 1 | \n| warn | `robots-txt` ,`sitemap` ,`llms-txt` missing at origin | 1 each | \n| info | `jsonld-types` : BlogPosting, Person, WebPage, FAQPage | 15 | \n\nEach finding shows the actual value, so you don't have to go and look:\n\n```\nOn-page\n  warn   meta-description        Meta description is 197 characters; expected 50-160.\n         url: https://bayuh85.github.io/atmorouter-blog/posts/applyfit/\n         evidence: {\"description\": \"A Python CLI and MCP server that keeps your profile and every job posting as plain files, scores fit offline from 0 to 100, ...\", \"length\": 197}\n```\n\nTwo of these deserve a comment, because a good audit tool should tell you things you then decide to ignore:\n\n`canonical-target`` robots-txt` / `sitemap` / `llms-txt`` bayuh85.github.io/robots.txt`), because that's the only place crawlers look. A project site under a subpath can't fix that without a user-site repo. Good to know before you blame the generator.\nThe real fixes are all in one Jekyll layout: shorten the `<title>` template, cap the description, add a default `og:image`. One PR, 15 pages fixed.\n\nFor comparison, the same command on `example.com` returns `score: 49/100` with no canonical, no `h1`, no description and no JSON-LD. Running `rankprobe robots https://www.nytimes.com` shows all nine AI crawlers blocked, which is a deliberate choice on their side. You probably want to know whether your site makes the same choice by accident.\n\nThen I asked for a plan on one post:\n\n```\nrankprobe plan https://bayuh85.github.io/atmorouter-blog/posts/vetskill/\nsent 5774 chars (~1444 tokens, estimated as chars/4) to atmo/deepseek-v4.1-flash at https://atmorouter.dev/v1\n\n# Fix plan — https://bayuh85.github.io/atmorouter-blog/posts/vetskill/\n\n## 1. robots-txt — create /robots.txt (also fixes sitemap discovery and locks in AI crawler access)\n...\n## 3. canonical-target — point the canonical at the URL you want indexed\n... If the dev.to post should stay canonical (you're treating GitHub Pages as a mirror), leave it ...\n\n## 4. title-length — shorten the title to 15–65 chars\nCurrent title is 170 characters ...\n<title>vetskill: scan a Claude Code skill before install</title>\n```\n\nThe prompt tells the model to work only from the findings it was given, and the first line tells you exactly how much was sent. 5.7 KB of JSON instead of the 26 KB HTML page. The plan took about 10 seconds.\n\nThe repo ships [`examples/github-actions-audit.yml`](https://github.com/bayuh85/rankprobe/blob/main/examples/github-actions-audit.yml). The important line is:\n\n```\n- run: rankprobe audit \"$TARGET\" --fail-under 80 --sarif rankprobe.sarif\n```\n\nPoint `TARGET` at your preview deploy URL, or at a built HTML file in the repo. A PR that adds a `noindex` or breaks the JSON-LD now fails the check, and the SARIF upload puts the findings in the code scanning tab. The audit only fetches your page plus robots.txt, the sitemap and llms.txt from its origin, and never calls a model.\n\n`plan` is one short chat call, so it should cost close to nothing. rankprobe defaults to [AtmoRouter](https://atmorouter.dev), a cheap AI API gateway and OpenRouter alternative: 181 models behind one OpenAI-compatible and Anthropic-compatible key, pay per token, averaging 79.8% below official prices when I checked today. The default model, `atmo/deepseek-v4.1-flash`, is free there, and that's the model in the run above.\n\nIt isn't tied to it. Any OpenAI-compatible endpoint works:\n\n```\nRANKPROBE_BASE_URL=http://127.0.0.1:11434/v1 RANKPROBE_MODEL=llama3.2 OPENAI_API_KEY=local \\\n  rankprobe plan https://your-site.dev\n```\n\n| Variable | Default | \n|---|---|\n| `ATMOROUTER_API_KEY` /`OPENAI_API_KEY` | first one set wins | \n| `RANKPROBE_BASE_URL` | `https://atmorouter.dev/v1` | \n| `RANKPROBE_MODEL` | `atmo/deepseek-v4.1-flash` | \n\nrankprobe reads the HTML the server sends. It doesn't run JavaScript, so a client-rendered SPA will look empty to it, and to a lot of crawlers too. It doesn't measure Core Web Vitals, check backlinks or know anything about rankings. The multi-page score adds penalties across pages, so a 15-page crawl bottoms out fast. Read the per-check counts, not the number. It's the pre-merge lint for SEO, not the whole SEO job.\n\n`plan`: Checks live in `checks.py`, each with an id, a severity and a message, so adding one is a small PR. If it catches a stray `noindex` before your launch, a star helps other people find it.", "url": "https://wpnews.pro/news/rankprobe-an-seo-and-ai-crawler-audit-that-fails-your-pr-before-a-stray-noindex", "canonical_source": "https://dev.to/bayuh85/rankprobe-an-seo-and-ai-crawler-audit-that-fails-your-pr-before-a-stray-noindex-ships-1h1g", "published_at": "2026-10-29 02:33:47+00:00", "updated_at": "2026-10-29 02:41:16.780910+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "developer-tools", "ai-search"], "entities": ["rankprobe", "GitHub Pages", "Jekyll", "GPTBot", "ClaudeBot", "PerplexityBot", "Google-Extended", "bayuh85"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/rankprobe-an-seo-and-ai-crawler-audit-that-fails-your-pr-before-a-stray-noindex", "markdown": "https://wpnews.pro/news/rankprobe-an-seo-and-ai-crawler-audit-that-fails-your-pr-before-a-stray-noindex.md", "text": "https://wpnews.pro/news/rankprobe-an-seo-and-ai-crawler-audit-that-fails-your-pr-before-a-stray-noindex.txt", "jsonld": "https://wpnews.pro/news/rankprobe-an-seo-and-ai-crawler-audit-that-fails-your-pr-before-a-stray-noindex.jsonld"}}