{"slug": "three-cloudflare-defaults-that-silently-broke-my-static-site-on-launch-day", "title": "Three Cloudflare defaults that silently broke my static site on launch day", "summary": "A developer shipping a static Astro site to Cloudflare Pages discovered three platform defaults that silently broke the site on launch day. Cloudflare's managed robots.txt injected AI crawler blocks, Pages added trailing slashes to URLs, and a sitemap filter broke due to path normalization, causing noindexed pages to be re-added. The developer advises deploying first and testing against the live origin rather than local builds.", "body_md": "I shipped a static Astro site to Cloudflare Pages this week. Build was green, every local check passed, the custom domain resolved. Three things were still broken, and all three were platform defaults I never touched.\n\nNone of them throw an error. That is the whole problem.\n\nMy repo has a four-line robots.txt. Allow everything, point at the sitemap. Here is what the live site actually served:\n\n```\n# BEGIN Cloudflare Managed content\nUser-agent: *\nContent-Signal: search=yes,ai-train=no,use=reference\nAllow: /\n\nUser-agent: Amazonbot\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: GPTBot\nDisallow: /\n...\n# END Cloudflare Managed Content\n\nUser-agent: *\nAllow: /\n```\n\nThat whole block is injected. It is the \"Managed robots.txt\" toggle under AI Crawl Control, on by default for a new zone. Googlebot and Bingbot still get through, so your search indexing looks fine and you have no reason to look.\n\nIf you actually want AI crawlers blocked, great, it did that for you. I wanted the opposite and had no idea it was happening. The toggle is at AI Crawl Control → Overview.\n\nCheck with one command, and read the whole body, not the status code:\n\n```\ncurl -s https://yoursite.com/robots.txt\n```\n\nI had configured `trailingSlash: 'never'`\n\n. Every canonical on the site pointed at the no-slash form. On Cloudflare Pages:\n\n```\nGET /bosses/how-many-bosses   →  308  →  /bosses/how-many-bosses/\n```\n\nPages adds the slash. So every canonical URL on 33 pages pointed at a URL that redirects. Not fatal, but it costs a hop on every indexable page, and the local static server I tested against did the opposite, which is why I never saw it.\n\nWorth knowing: this is a host behavior, not a framework one. You cannot settle it locally. Deploy one page and curl it.\n\nThis is the one I actually enjoyed.\n\nMy sitemap filter excluded noindex pages by exact path match, built from filesystem routes, so `/tags/foo`\n\n. After the switch, `url.pathname`\n\nbecame `/tags/foo/`\n\n. The lookup stopped matching. Nothing errored. The sitemap went from 33 URLs to 44, quietly re-adding 11 pages that still carried `noindex`\n\nin their HTML.\n\nA sitemap that lists noindexed pages is a contradiction you send Google on purpose. Fix is a one-liner, normalise before comparing:\n\n``` js\nconst p = decodeURIComponent(new URL(url).pathname).replace(/\\/+$/, '') || '/';\nreturn !noindexPaths.has(p);\n```\n\nThe lesson is not \"normalise your paths\". It is that a config change in one file silently changed the contract for a lookup in a different file, and both were \"working\" the entire time.\n\nNew zone, default off. `http://`\n\nserves a 200 instead of redirecting, and Search Console will happily index the http version as a separate property splitting your signals. SSL/TLS → Edge Certificates → Always Use HTTPS.\n\nI had a local production build, a header check, a link checker, and a sitemap test. All green. Every one of these three lived in the gap between \"my build output\" and \"what the host actually serves\".\n\nSo: deploy first, then run your checks against the live origin. Not the build directory. Not localhost. The origin.\n\nThe site was [gawrguraquestforbread.com](https://gawrguraquestforbread.com/), a small fan wiki, if you want to see what the fixed output looks like.", "url": "https://wpnews.pro/news/three-cloudflare-defaults-that-silently-broke-my-static-site-on-launch-day", "canonical_source": "https://dev.to/hblai_filmlook/three-cloudflare-defaults-that-silently-broke-my-static-site-on-launch-day-132g", "published_at": "2026-08-24 02:17:24+00:00", "updated_at": "2026-08-24 02:43:19.577209+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Cloudflare Pages", "Astro", "Googlebot", "Bingbot", "Search Console"], "alternates": {"html": "https://wpnews.pro/news/three-cloudflare-defaults-that-silently-broke-my-static-site-on-launch-day", "markdown": "https://wpnews.pro/news/three-cloudflare-defaults-that-silently-broke-my-static-site-on-launch-day.md", "text": "https://wpnews.pro/news/three-cloudflare-defaults-that-silently-broke-my-static-site-on-launch-day.txt", "jsonld": "https://wpnews.pro/news/three-cloudflare-defaults-that-silently-broke-my-static-site-on-launch-day.jsonld"}}