{"slug": "getting-a-brand-new-site-crawled-who-actually-shows-up-and-in-what-order", "title": "Getting a brand new site crawled: who actually shows up, and in what order", "summary": "A developer launched a static site on 15 October and logged 9,987 crawler requests over the following 15 days, finding that Anthropic's ClaudeBot and OpenAI's GPTBot and OAI-SearchBot arrived within roughly 65-70 minutes of the first page going live, while Googlebot made zero requests for four days until the domain was verified and the sitemap submitted in Search Console, after which it appeared within six minutes. The log also showed ClaudeBot fetching robots.txt once per crawl session (78 times in 15 days) while GPTBot requested it zero times across 391 fetches, and that GoogleOther fetched nearly as much as Googlebot (520 vs 541 requests).", "body_md": "We put a static site online on 15 October, published nothing for four days, then published three articles in one evening. The access log was recording the whole time. Fifteen days later it holds 9,987 requests, and the order in which the internet noticed us is not the order anybody plans for.\n\nNo theory below. Every number is from that log.\n\nThe first page went live at 13:14 UTC. The log of what happened next, in order of first appearance:\n\n| Crawler | First seen | Requests, week 1 | Requests, 15 days | \n|---|---|---|---|\n| ClaudeBot | 15.10, 65 min after launch | 226 | 484 | \n| GPTBot | 15.10, 70 min after launch | 210 | 391 | \n| OAI-SearchBot | 15.10, 70 min after launch | 33 | 108 | \n| bingbot | 16.10 | 70 | 164 | \n| Googlebot | 19.10, 19:41 | 328 | 541 | \n| GoogleOther | 19.10, 23:01 | 281 | 520 | \n| AhrefsBot | 20.10 | 91 | 202 | \n| ChatGPT-User | 21.10 | 4 | 39 | \n\nNothing linked to the site yet. No submission had been made anywhere. Anthropic's and OpenAI's crawlers found a brand new hostname on a small VPS within the hour, and started walking it.\n\nThat is the practical point: **if you publish technical writing today, your first readers are retrieval pipelines, and they arrive before any search engine knows you exist.** Decide your `robots.txt` policy before you publish, not after you notice.\n\nNote what ClaudeBot did in its very first two requests, one second apart: `/robots.txt`, then `/`. It has asked for `robots.txt` 78 times in fifteen days, once per crawl session. GPTBot has requested it zero times in the same window, while fetching 391 URLs. Same two seconds of effort, opposite behaviour. We will come back to that in a separate piece, because it decides which blocking method is worth deploying.\n\nZero Googlebot requests in the first four days. Not throttled, not slow: **absent**. The site was reachable, had a sitemap, had `robots.txt`, and was linked from an established domain we also run.\n\nWhat changed it was verifying the domain in Search Console and submitting the sitemap. From the log, that evening:\n\n`/robots.txt` and `/sitemap.xml`, in the same second.\n**Six minutes.** You can argue about whether Search Console is formally required - it is not - but the difference between four days of silence and six minutes is hard to ignore when you are starting from nothing.\n\nUse a **domain property** rather than a URL-prefix property if you can add a DNS record: one TXT at the apex covers every subdomain you will ever add, and it survives an http/https or www change that would otherwise split your data across properties.\n\nThen, three hours and twenty minutes after Googlebot, **GoogleOther** showed up and has since fetched almost as much as Googlebot itself: 520 requests against 541. It is a separate crawler using the same infrastructure for non-search product work. If you ever decide to restrict Google, know that it is two names, not one.\n\nIndexNow is a simple deal: you host a key file, then POST a list of URLs you changed, and participating engines (Bing, Yandex, Seznam, Naver) are told immediately instead of waiting for a crawl.\n\nThe whole protocol is one request:\n\n```\ncurl -X POST https://api.indexnow.org/indexnow \\\n  -H \"Content-Type: application/json; charset=utf-8\" \\\n  -d '{\n    \"host\": \"example.com\",\n    \"key\": \"YOUR-KEY\",\n    \"keyLocation\": \"https://example.com/YOUR-KEY.txt\",\n    \"urlList\": [\"https://example.com/new-post/\"]\n  }'\n```\n\nThe key is a file at your domain root whose name is the key and whose content is the same string. That is the entire verification: if you can put a file there, you own the host.\n\nA `200` means accepted, not indexed. But bingbot showed up on its own the day after our first submission, without an account, a console or a verification step - three days before Googlebot. For a new site that is a meaningful head start, and it costs one line in the deploy pipeline.\n\nHere is the part that nearly fooled us, and it is the most useful paragraph in this post.\n\nMultilingual Hugo generates a **sitemap index** at `/sitemap.xml` - a list of other sitemaps, one per language - rather than a list of pages. We submitted that, and Search Console reported:\n\nStatus: Success. Discovered pages: 0.\n\nBoth halves are true and they look contradictory. The index was fetched and parsed correctly, so the status is a success; the index itself contains no page URLs, so the discovered count stays at zero until the children are fetched separately. A quick glance at \"Success\" and you walk away believing the job is done.\n\nWe submitted the child sitemap for the main language as well:\n\nStatus: Success. Discovered pages: 23.\n\nSo: **if your sitemap is an index, submit the per-language children too**, and treat the discovered-pages count, not the status word, as the thing that tells you it worked. The same applies to any generator that emits an index - large WordPress installs with Yoast do, and so does anything paginating its sitemaps past 50,000 URLs.\n\nFor eleven days the log had eight or nine distinct crawlers in it. Then, between 26 and 29 October, fourteen new ones appeared within four days:\n\nElevenlabsCrawler (448 requests), meta-externalagent (322), MJ12bot (319), AwarioBot (308), SofyaBot (232), SemrushBot (213), Amazonbot (96), PerplexityBot (63), DataForSeoBot (62), YandexBot (49), Applebot (30), archive.org_bot (26), Baiduspider (14), DuckDuckBot (8).\n\nThe pattern is clean enough to plan around. SEO backlink crawlers, brand-monitoring crawlers and the second tier of AI crawlers do not discover sites themselves - they read other people's indexes. Once Google and Bing have you, you are in a feed that resells discovery, and the rest follow within a fortnight whether you wanted them or not.\n\nWhat that costs, measured: of 153.4 MB served in fifteen days, **88.8 MB went to user agents that identify themselves as crawlers**. That is 58% of all bytes, against roughly 4,000 browser-like requests. On a static site served from a tiny VPS, nobody notices. On anything that renders pages per request, that is your capacity plan.\n\nFor a new site, in this order, with the reason each step exists:\n\n`robots.txt` honest.\nSubmit the sitemap to Search Console on day one, before any content exists. Verification is a DNS record and takes five minutes; doing it early means the crawler already knows the site when the first article appears, instead of starting the clock after it. We spent four days publishing into a void we could have avoided.\n\nThe IndexNow ping we did fix: it now runs as a step in the deploy workflow, in the same place as the [post-deploy smoke test](https://news.rozoom.app/self-hosting/github-actions-rsync-deploy-no-shell/), so a publish that reaches the server always reaches the engines too. Doing it by hand means forgetting it exactly once, on the article that mattered.\n\nIt is one site, one niche, one fifteen-day window, on a hostname with no history. Crawl budget for an established domain behaves differently. We also cannot separate \"arrived because of Search Console\" from \"arrived because it was the fourth day\" with a sample of one - the six-minute gap is suggestive, not a controlled experiment. And `robots.txt` fetch counts measure fetches, not obedience: a crawler that reads the file and ignores it looks identical to a polite one in this data.\n\n*Originally published at [news.rozoom.app](https://news.rozoom.app/self-hosting/getting-a-new-site-crawled/).*", "url": "https://wpnews.pro/news/getting-a-brand-new-site-crawled-who-actually-shows-up-and-in-what-order", "canonical_source": "https://dev.to/rozoomnews/getting-a-brand-new-site-crawled-who-actually-shows-up-and-in-what-order-4fc", "published_at": "2026-10-30 16:39:22+00:00", "updated_at": "2026-10-30 16:51:35.354754+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search"], "entities": ["ClaudeBot", "GPTBot", "OAI-SearchBot", "Googlebot", "GoogleOther", "Anthropic", "OpenAI", "IndexNow"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/getting-a-brand-new-site-crawled-who-actually-shows-up-and-in-what-order", "markdown": "https://wpnews.pro/news/getting-a-brand-new-site-crawled-who-actually-shows-up-and-in-what-order.md", "text": "https://wpnews.pro/news/getting-a-brand-new-site-crawled-who-actually-shows-up-and-in-what-order.txt", "jsonld": "https://wpnews.pro/news/getting-a-brand-new-site-crawled-who-actually-shows-up-and-in-what-order.jsonld"}}