This series is written in the open, from a real production system. This chapter is the reach stack: prerendering, robots.txt, llms.txt, IndexNow, and measuring it all with access logs. [All chapters and diagrams live in the public repo.]
Somewhere between "launched" and "learned people exist," every product
discovers distribution. Ours arrived with a twist that defines this
chapter: half your future traffic now arrives via machines that never render your app — search crawlers that mostly cope, and AI answer
The FoxyInvoice app is an Angular SPA: the server ships an empty shell
and JavaScript builds the page. Human browsers: fine. Googlebot:
tolerant, eventually. AI crawlers: blind. Fetch the marketing page
with curl — which is exactly what a crawler does — and you got:
<title>Invoicing</title>
<app-root></app-root>
One word and an empty div. To every machine reader, our entire free
invoice-template business was the word "Invoicing."
Our robots.txt doesn't list what's forbidden — it lists what's
allowed, then disallows everything else:
User-agent: *
Allow: /$ /pricing /privacy /terms /templates /templates/*
Disallow: /
Every future route — admin consoles, /upgrade, QA harnesses — is
non-indexable by default until deliberately made public. The same
policy is enforced server-side with X-Robots-Tag: noindex, nofollow
headers at the edge proxy, per URL: private pages carry it; public
pages don't. Belt and suspenders, because robots.txt disallow alone
never guarantees de-indexing.
Full server-side rendering means an always-on SSR server — violating
our boringness constraint — for pages that are 95% app shell. Instead,
a post-build script generates static, crawler-ready copies of every
public route when the SPA compiles:
<title> and meta description per page ("Free Plumbing Invoice
Template — FoxyInvoice")
The edge proxy's try_files serves these static files to anything that
fetches the URL; real browsers still get the app (Angular replaces the
static body on boot). No new server, no SSR framework, crawler-complete
pages.
llms.txt — a menu for answer engines
The emerging convention: a markdown file at the root telling AI systems
what the product is and linking its key pages, in their vocabulary.
Ours lists the product, every template page with a one-line
description, and contact points. Cheap, human-readable, and exactly
the artifact a "recommend an invoicing tool" query wants.
Crawlers traditionally rediscover content on their own schedule.
IndexNow flips it: on every deploy, the pipeline POSTs all sitemap
URLs to the alliance endpoint (Bing-powered — which feeds several AI
answer engines) with a key file proving domain ownership. Our first
ping returned HTTP 202 — accepted — and the pages were in Bing's queue
the same evening.
"You can't improve what you can't see," so the edge now logs every
request as JSON (rotated, bounded). The analysis is one script:
user-agents, paths, crawler classes. Within days we could see search
crawlers reading robots.txt and template pages — and, just as
valuable, probe-noise (stray /wp-admin scans) being correctly
absorbed. Pair this with Google Search Console + Bing Webmaster
registration (a five-minute runbook lives in the repo) for the
indexing-and-impressions view logs can't give you.
Reach is only worth building if it lands somewhere. The template
gallery — now crawlable — is the top of a three-slice funnel we shipped
in order:
Each slice was verified in a real browser before shipping — the funnel
is the one place where "should work" is banned.
Recap. Serve machines real HTML (prerendered at build), allowlist
what's public (default-closed), publish llms.txt, push updates via
IndexNow, and measure with access logs + Search Console. The era of
"GEO" — being the machine-readable answer — is just SEO where the
reader never renders your JavaScript.
Reading this and want to be found the way this series was? Create a free workspace at
foxyinvoice.com, then redeem founding code
U8B4Z8S87X on the Upgrade page — 6 months of Pro, free, no card. If anything
breaks, there's a feedback button in the app. I read every one.
Next: Chapter 12 — Engagement & automation: UI first, then automate.