I shipped a static Astro site to Cloudflare Pages this week. Build was green, every local check passed, the custom domain resolved. Three things were still broken, and all three were platform defaults I never touched.
None of them throw an error. That is the whole problem.
My repo has a four-line robots.txt. Allow everything, point at the sitemap. Here is what the live site actually served:
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
...
User-agent: *
Allow: /
That whole block is injected. It is the "Managed robots.txt" toggle under AI Crawl Control, on by default for a new zone. Googlebot and Bingbot still get through, so your search indexing looks fine and you have no reason to look.
If you actually want AI crawlers blocked, great, it did that for you. I wanted the opposite and had no idea it was happening. The toggle is at AI Crawl Control → Overview.
Check with one command, and read the whole body, not the status code:
curl -s https://yoursite.com/robots.txt
I had configured trailingSlash: 'never'
. Every canonical on the site pointed at the no-slash form. On Cloudflare Pages:
GET /bosses/how-many-bosses → 308 → /bosses/how-many-bosses/
Pages adds the slash. So every canonical URL on 33 pages pointed at a URL that redirects. Not fatal, but it costs a hop on every indexable page, and the local static server I tested against did the opposite, which is why I never saw it.
Worth knowing: this is a host behavior, not a framework one. You cannot settle it locally. Deploy one page and curl it.
This is the one I actually enjoyed.
My sitemap filter excluded noindex pages by exact path match, built from filesystem routes, so /tags/foo
. After the switch, url.pathname
became /tags/foo/
. The lookup stopped matching. Nothing errored. The sitemap went from 33 URLs to 44, quietly re-adding 11 pages that still carried noindex
in their HTML.
A sitemap that lists noindexed pages is a contradiction you send Google on purpose. Fix is a one-liner, normalise before comparing:
const p = decodeURIComponent(new URL(url).pathname).replace(/\/+$/, '') || '/';
return !noindexPaths.has(p);
The lesson is not "normalise your paths". It is that a config change in one file silently changed the contract for a lookup in a different file, and both were "working" the entire time.
New zone, default off. http://
serves a 200 instead of redirecting, and Search Console will happily index the http version as a separate property splitting your signals. SSL/TLS → Edge Certificates → Always Use HTTPS.
I had a local production build, a header check, a link checker, and a sitemap test. All green. Every one of these three lived in the gap between "my build output" and "what the host actually serves".
So: deploy first, then run your checks against the live origin. Not the build directory. Not localhost. The origin.
The site was gawrguraquestforbread.com, a small fan wiki, if you want to see what the fixed output looks like.