cd /news/ai-crawlers/i-documented-my-static-site-s-cache-… · home › topics › ai-crawlers › article
[ARTICLE · art-173937] src=dev.to ↗ pub= topic=ai-crawlers verified=true sentiment=↓ negative

I Documented My Static Site's Cache Headers. curl -I Says Otherwise.

A developer documented that their Firebase Hosting cache rules for a static Next.js site never worked as published: the HTML rule's glob matched request paths ending in .html, but cleanUrls and trailingSlash meant no real request ever used that form, so pages, llms.txt and sitemap.xml were all served with Firebase's default max-age=3600 instead of max-age=0, must-revalidate. A second rule intended for content-hashed bundles instead matched by file extension, leaving the headshot and site icon cached as immutable for a year. The developer traced both mismatches to verifying the config file rather than the live response headers via curl -I.

recommendation

Verify cache headers with curl -I against live URLs after every deploy, and scope immutable caching to hashed asset folders rather than file extensions.

more in playbook: Running AI in production
by read7 min views3 publishedNov 21, 2026

A few weeks ago I wrote up how my personal site is built for search engines and AI crawlers: a static Next.js export, one typed registry of posts that drives everything, Article and FAQ structured data, an llms.txt, IndexNow on deploy, and a caching policy on Firebase Hosting. I described that caching policy with some confidence. HTML gets max-age=0, must-revalidate, hashed assets get cached for a year.

Then I ran curl -I against the live site, which is something I should have done before writing the sentence.

for u in / /blog/developer-blog-seo-ai-search-nextjs-static/ /llms.txt /sitemap.xml; do
  curl -sI "https://nabeelbaghoor.com$u" | grep -i cache-control
done

Every one of them came back cache-control: max-age=3600. Not the header I wrote. Not the header in my config. Firebase's default.

This post is about why, and about the second mismatch I found while I was in there, because both come from the same habit: I verified the config file and never verified the output.

Here is the relevant part of firebase.json, unchanged since the site's first commit in July:

{
  "hosting": {
    "public": "out",
    "cleanUrls": true,
    "trailingSlash": true,
    "headers": [
      {
        "source": "**/*.@(js|css|svg|jpg|jpeg|png|woff2)",
        "headers": [{ "key": "Cache-Control", "value": "public, max-age=31536000, immutable" }]
      },
      {
        "source": "**/*.@(html)",
        "headers": [{ "key": "Cache-Control", "value": "public, max-age=0, must-revalidate" }]
      }
    ]
  }
}

Read it as a file list and it is fine. The export writes out/blog/some-post/index.html, and index.html ends in .html, so the second rule covers it.

But the source glob is matched against the request path, not against the file that ends up being served. With cleanUrls and trailingSlash on, nobody ever requests a path ending in .html. Readers and crawlers ask for /blog/some-post/. If you do ask for the .html file directly, Firebase answers with a 301 to the clean URL:

curl -sI https://nabeelbaghoor.com/blog/developer-blog-seo-ai-search-nextjs-static/index.html

So the HTML rule has never matched a single real request. Every page, plus llms.txt, sitemap.xml and the small .txt payloads Next.js uses for client-side navigation, has been served with Firebase's default one hour. For a blog, one hour of staleness after a deploy is honestly not a disaster. What bothers me is that I published the wrong number as a recommendation, with my own site as the example.

The first rule has the opposite problem. It does match, and it matches more than I meant.

The reasoning behind immutable for a year is that the files have content hashes in their names, so a new version gets a new URL. That is true for everything under /_next/static/. It is not true for files I put in public/ myself:

curl -sI https://nabeelbaghoor.com/assets/nabeel.jpeg | grep -i cache-control
curl -sI https://nabeelbaghoor.com/icon.svg | grep -i cache-control

That is my headshot and the site icon. If I replace either one and keep the filename, a returning visitor's browser is allowed to keep showing the old one for a year without asking. The glob was written for hashed bundles and I pointed it at file extensions instead of at the folder that actually holds hashed files. While checking this I also noticed the export still ships the starter next.svg, vercel.svg, file.svg, globe.svg and window.svg from create-next-app, also cached as immutable. Harmless, but it tells you how closely I had looked at out/.

The same article says the framework, the host and the canonical tags must agree on one URL form, or you get redirect chains. My site uses trailing slashes. So I checked what the canonical tag says:

curl -s https://nabeelbaghoor.com/blog/developer-blog-seo-ai-search-nextjs-static/ \
  | grep -o '<link rel="canonical"[^>]*>'

Trailing slash, correct. Good. Then I looked at the code that produces it:

const url = `${BLOG_SITE.url}/blog/${post.slug}`;
return {
  alternates: { canonical: url },
  openGraph: { url, /* ... */ },
};

No trailing slash. The tag comes out right only because Next.js, as far as I can tell from the output, applies the trailingSlash: true setting when it renders metadata. That normalisation is a framework behaviour I was leaning on without knowing it, and it only applies to the metadata API. Every other place that builds the same string by hand gets nothing:

app/sitemap.ts maps over the registry with `https://nabeelbaghoor.com/blog/${p.slug}`. The live sitemap has 116 <loc> entries, and every one except the homepage, all 114 posts plus /blog, answers with a 301 to the slash version. mainEntityOfPage is the no-slash URL, so the structured data points at a redirect while the canonical tag next to it points at the real page.llms.txt. None of this breaks anything you can see. Crawlers follow a single 301. But a sitemap is supposed to list the URLs you want indexed, not URLs that redirect to them, and I wrote a whole section saying exactly that.

The irony is that the post's main point was "one typed registry, everything derived from it, so nothing can drift". The registry did its job: 114 entries, 114 markdown files, no orphans either way. What drifted was the thing I built from the registry in four places, the URL, with four copies of the same template string and only one of them quietly fixed by the framework.

I am writing this before changing anything, partly because I want the before state on record and partly because the URL fix is the boring one-character kind I should make carefully.

One URL function. The slug-to-URL rule belongs in the registry module, next to the registry:

export function postUrl(slug: string): string {
  return `${BLOG_SITE.url}/blog/${slug}/`;
}

Then the metadata, the sitemap, the JSON-LD and llms.txt all call it, and the trailing slash is decided once, in the same file as the data it describes.

Header rules keyed on what the browser asks for. The immutable rule should target /_next/static/**, where filenames really are content hashed. Images and icons in public/ get a short cache. Pages need a rule that matches clean paths rather than .html. Firebase decides which header wins when more than one rule matches a path, and I am not going to describe that precedence from memory a second time. The config change is only finished when the curl below says it is.

A check that reads the deployed site, not the repo. This is the real lesson, so it is the piece I care most about:

// check-headers.mjs, run after every deploy
const SITE = "https://nabeelbaghoor.com";
const expect = [
  ["/", /max-age=0/],
  ["/blog/", /max-age=0/],
  ["/llms.txt", /max-age=0/],
  ["/assets/nabeel.jpeg", /^(?!.*immutable)/],
];

let failed = 0;
for (const [path, rule] of expect) {
  const res = await fetch(SITE + path, { method: "HEAD", redirect: "manual" });
  const cc = res.headers.get("cache-control") ?? "";
  const ok = res.status === 200 && rule.test(cc);
  if (!ok) failed++;
  console.log(ok ? "ok  " : "FAIL", res.status, path, cc);
}

const xml = await (await fetch(SITE + "/sitemap.xml")).text();
for (const [, loc] of xml.matchAll(/<loc>([^<]+)<\/loc>/g)) {
  const res = await fetch(loc, { method: "HEAD", redirect: "manual" });
  if (res.status !== 200) { failed++; console.log("FAIL", res.status, loc); }
}
process.exit(failed ? 1 : 0);

Today that script would fail all four checks in its list and 115 of the 116 sitemap entries. That is the right result, and it is the whole argument for writing it.

This is the same shape as a few other things I have written about recently. My cron job returned 200 when every fetch failed, and my blog's internal links only pointed backward in time. Each time the code or the config looked correct when I read it, and the only thing that showed the problem was asking the running system what it actually did.

For a static site, that is cheap. The whole site is a folder of files behind a CDN. curl -I costs nothing, and a fifteen-line script after firebase deploy would have caught both of these in July. I verified the build because the build has a type checker. I never verified the deploy because the deploy has no opinion.

The rest of the original article still stands. The static export, the typed registry, FAQ schema that matches a visible FAQ, llms.txt and the writing advice all hold up. One correction on IndexNow while I am being thorough: the ping after deploy lives in the instructions of the scheduled routine that publishes my posts, not in npm run deploy. A manual deploy pings nothing. That is fine as long as I know it, and now it is written down.

The original walkthrough is on my site: How I built my developer blog for Google and AI search with a static Next.js export. I will update it once the fixes are live and the script passes.

If you host a Next.js export on Firebase, Netlify or Cloudflare Pages: have you checked what cache header your pages are actually served with? I would be curious how many of us are on the platform default without knowing it.

── more in #ai-crawlers 4 stories · sorted by recency
── more on @firebase hosting 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/i-documented-my-stat…] indexed:0 read:7min 2026-11-21 · —