Getting a brand new site crawled: who actually shows up, and in what order

We put a static site online on 15 October, published nothing for four days, then published three articles in one evening. The access log was recording the whole time. Fifteen days later it holds 9,987 requests, and the order in which the internet noticed us is not the order anybody plans for.

No theory below. Every number is from that log.

The first page went live at 13:14 UTC. The log of what happened next, in order of first appearance:

Nothing linked to the site yet. No submission had been made anywhere. Anthropic's and OpenAI's crawlers found a brand new hostname on a small VPS within the hour, and started walking it.

That is the practical point: if you publish technical writing today, your first readers are retrieval pipelines, and they arrive before any search engine knows you exist. Decide your robots.txt policy before you publish, not after you notice.

Note what ClaudeBot did in its very first two requests, one second apart: /robots.txt, then /. It has asked for robots.txt 78 times in fifteen days, once per crawl session. GPTBot has requested it zero times in the same window, while fetching 391 URLs. Same two seconds of effort, opposite behaviour. We will come back to that in a separate piece, because it decides which blocking method is worth deploying.

Zero Googlebot requests in the first four days. Not throttled, not slow: absent. The site was reachable, had a sitemap, had robots.txt, and was linked from an established domain we also run.

What changed it was verifying the domain in Search Console and submitting the sitemap. From the log, that evening:

/robots.txt and /sitemap.xml, in the same second. Six minutes. You can argue about whether Search Console is formally required - it is not - but the difference between four days of silence and six minutes is hard to ignore when you are starting from nothing.

Use a domain property rather than a URL-prefix property if you can add a DNS record: one TXT at the apex covers every subdomain you will ever add, and it survives an http/https or www change that would otherwise split your data across properties.

Then, three hours and twenty minutes after Googlebot, GoogleOther showed up and has since fetched almost as much as Googlebot itself: 520 requests against 541. It is a separate crawler using the same infrastructure for non-search product work. If you ever decide to restrict Google, know that it is two names, not one.

IndexNow is a simple deal: you host a key file, then POST a list of URLs you changed, and participating engines (Bing, Yandex, Seznam, Naver) are told immediately instead of waiting for a crawl.

The whole protocol is one request:

curl -X POST https://api.indexnow.org/indexnow \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "example.com",
    "key": "YOUR-KEY",
    "keyLocation": "https://example.com/YOUR-KEY.txt",
    "urlList": ["https://example.com/new-post/"]
  }'

The key is a file at your domain root whose name is the key and whose content is the same string. That is the entire verification: if you can put a file there, you own the host.

A 200 means accepted, not indexed. But bingbot showed up on its own the day after our first submission, without an account, a console or a verification step - three days before Googlebot. For a new site that is a meaningful head start, and it costs one line in the deploy pipeline.

Here is the part that nearly fooled us, and it is the most useful paragraph in this post.

Multilingual Hugo generates a sitemap index at /sitemap.xml - a list of other sitemaps, one per language - rather than a list of pages. We submitted that, and Search Console reported:

Status: Success. Discovered pages: 0.

Both halves are true and they look contradictory. The index was fetched and parsed correctly, so the status is a success; the index itself contains no page URLs, so the discovered count stays at zero until the children are fetched separately. A quick glance at "Success" and you walk away believing the job is done.

We submitted the child sitemap for the main language as well:

Status: Success. Discovered pages: 23.

So: if your sitemap is an index, submit the per-language children too, and treat the discovered-pages count, not the status word, as the thing that tells you it worked. The same applies to any generator that emits an index - large WordPress installs with Yoast do, and so does anything paginating its sitemaps past 50,000 URLs.

For eleven days the log had eight or nine distinct crawlers in it. Then, between 26 and 29 October, fourteen new ones appeared within four days:

ElevenlabsCrawler (448 requests), meta-externalagent (322), MJ12bot (319), AwarioBot (308), SofyaBot (232), SemrushBot (213), Amazonbot (96), PerplexityBot (63), DataForSeoBot (62), YandexBot (49), Applebot (30), archive.org_bot (26), Baiduspider (14), DuckDuckBot (8).

The pattern is clean enough to plan around. SEO backlink crawlers, brand-monitoring crawlers and the second tier of AI crawlers do not discover sites themselves - they read other people's indexes. Once Google and Bing have you, you are in a feed that resells discovery, and the rest follow within a fortnight whether you wanted them or not.

What that costs, measured: of 153.4 MB served in fifteen days, 88.8 MB went to user agents that identify themselves as crawlers. That is 58% of all bytes, against roughly 4,000 browser-like requests. On a static site served from a tiny VPS, nobody notices. On anything that renders pages per request, that is your capacity plan.

For a new site, in this order, with the reason each step exists:

robots.txt honest. Submit the sitemap to Search Console on day one, before any content exists. Verification is a DNS record and takes five minutes; doing it early means the crawler already knows the site when the first article appears, instead of starting the clock after it. We spent four days publishing into a void we could have avoided.

The IndexNow ping we did fix: it now runs as a step in the deploy workflow, in the same place as the post-deploy smoke test, so a publish that reaches the server always reaches the engines too. Doing it by hand means forgetting it exactly once, on the article that mattered.

It is one site, one niche, one fifteen-day window, on a hostname with no history. Crawl budget for an established domain behaves differently. We also cannot separate "arrived because of Search Console" from "arrived because it was the fourth day" with a sample of one - the six-minute gap is suggestive, not a controlled experiment. And robots.txt fetch counts measure fetches, not obedience: a crawler that reads the file and ignores it looks identical to a polite one in this data.

Originally published at news.rozoom.app.