{"slug": "show-hn-fetchcheck-send-a-signed-crawler-to-any-site-and-see-if-it-gets-in", "title": "Show HN: Fetchcheck – send a signed crawler to any site and see if it gets in", "summary": "Fetchcheck, a new tool from Sitedex, sends a single signed request from its SitedexBot crawler to any URL and reports whether the site allows, blocks, challenges, or returns empty content, helping site owners verify that trusted AI crawlers can access their pages. The tool respects robots.txt, never stores or republishes pages, and uses the open Web Bot Auth standard for signing requests. It checks one URL per run with up to four requests, including an unsigned retry to compare how the site treats signed versus unsigned crawlers.", "body_md": "fetchcheck\n\n# See how any site handles a signed crawler.\n\nSitedexBot is our crawler. It signs every request so a site can verify it's really us. Paste a URL and we send one signed request to the page, then show you exactly what came back: allowed, blocked, challenged, or a page with nothing in it.\n\nEnter any domain. We check `robots.txt`\n\n, then fetch the page as SitedexBot. We don't keep your page. Don't paste URLs with secrets in them — the address is what we check.\n\n## Why check whether a signed crawler gets in\n\n**Bot protection is on by default now.** WAFs and bot-management rules block or challenge automated traffic before it reaches your server. That is good for stopping scrapers and abuse. It also catches the crawlers you may actually want to let in.\n\nWhen someone asks an AI assistant about your company, product, or docs, it often fetches your pages to answer. If your site turns that request away, or hands it a blank page with no content, the assistant answers from whatever else it can find instead of what you published.\n\n**A signed crawler proves who it is on every request,** using an open standard that large networks already recognize. That lets you allow the crawlers you trust by name instead of blocking everything. fetchcheck sends one signed request from SitedexBot and shows you the result, so you can see whether a trusted crawler gets in, and whether your content is actually there when it does.\n\n## What fetchcheck does\n\n- Sends one signed\n`GET`\n\nfrom SitedexBot, our crawler. Every request is signed so a site can verify it is really us. - Respects\n`robots.txt`\n\n. If your rules tell SitedexBot not to fetch a page, we don't. We report the rule instead. - Never stores or republishes your page. We show the response status, the headers, and whether the HTML had readable content. We don't keep the page.\n- Checks one URL per run, using up to four requests: your\n`robots.txt`\n\n, the page (signed, as SitedexBot), and — only if that signed request is turned away — one unsigned retry for comparison. That retry comes from our own infrastructure, not a real browser or a home connection; it simply drops the signature, so we can see whether the refusal is aimed at signed or identified crawlers specifically. And if`https`\n\ndoesn't answer at all, one plain`http`\n\ncheck, to tell a site that is down from one that only answers over`http`\n\n.\n\nSitedexBot signs its requests using [Web Bot Auth](https://sitedex.dev/bot), an open standard for identifying trusted crawlers. Our signing keys are public. More: [sitedex.dev/bot](https://sitedex.dev/bot).\n\nChecking your own signing-key setup instead? [Try keycheck, the Web Bot Auth key check](/tools/keycheck).", "url": "https://wpnews.pro/news/show-hn-fetchcheck-send-a-signed-crawler-to-any-site-and-see-if-it-gets-in", "canonical_source": "https://sitedex.dev/tools/fetchcheck", "published_at": "2026-08-03 20:46:36+00:00", "updated_at": "2026-08-03 20:52:22.965351+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure"], "entities": ["Fetchcheck", "Sitedex", "SitedexBot", "Web Bot Auth"], "alternates": {"html": "https://wpnews.pro/news/show-hn-fetchcheck-send-a-signed-crawler-to-any-site-and-see-if-it-gets-in", "markdown": "https://wpnews.pro/news/show-hn-fetchcheck-send-a-signed-crawler-to-any-site-and-see-if-it-gets-in.md", "text": "https://wpnews.pro/news/show-hn-fetchcheck-send-a-signed-crawler-to-any-site-and-see-if-it-gets-in.txt", "jsonld": "https://wpnews.pro/news/show-hn-fetchcheck-send-a-signed-crawler-to-any-site-and-see-if-it-gets-in.jsonld"}}