{"slug": "building-foxyinvoice-chapter-11-reach-seo-ai-crawlers-and-being-the-machine", "title": "Building FoxyInvoice — Chapter 11: Reach — SEO, AI crawlers, and being the machine-readable answer", "summary": "A developer building the FoxyInvoice invoicing app documented a distribution stack that serves crawler-ready static HTML for its Angular SPA, publishes an llms.txt file for AI answer engines, and pushes sitemap updates via IndexNow. The build-time prerendering script generates per-route titles and meta descriptions, while robots.txt allowlists public routes and edge proxy headers enforce noindex on private ones. Access logs analyzed by a script confirmed search crawlers reading robots.txt and template pages within days of launch.", "body_md": "*This series is written in the open, from a real production system. This chapter is the reach stack: prerendering, robots.txt, llms.txt, IndexNow, and measuring it all with access logs. [All chapters and diagrams live in the public repo.]*\n\nSomewhere between \"launched\" and \"learned people exist,\" every product\n\ndiscovers distribution. Ours arrived with a twist that defines this\n\nchapter: **half your future traffic now arrives via machines that never render your app** — search crawlers that mostly cope, and AI answer\n\nThe FoxyInvoice app is an Angular SPA: the server ships an empty shell\n\nand JavaScript builds the page. Human browsers: fine. Googlebot:\n\ntolerant, eventually. **AI crawlers: blind.** Fetch the marketing page\n\nwith `curl` — which is exactly what a crawler does — and you got:\n\n```\n<title>Invoicing</title>\n<app-root></app-root>\n```\n\nOne word and an empty div. To every machine reader, our entire free\n\ninvoice-template business was the word \"Invoicing.\"\n\nOur `robots.txt` doesn't list what's forbidden — it lists what's\n\n*allowed*, then disallows everything else:\n\n```\nUser-agent: *\nAllow: /$ /pricing /privacy /terms /templates /templates/*\nDisallow: /\n```\n\nEvery future route — admin consoles, `/upgrade`, QA harnesses — is\n\n**non-indexable by default** until deliberately made public. The same\n\npolicy is enforced *server-side* with `X-Robots-Tag: noindex, nofollow`\n\nheaders at the edge proxy, per URL: private pages carry it; public\n\npages don't. Belt and suspenders, because robots.txt disallow alone\n\nnever guarantees de-indexing.\n\nFull server-side rendering means an always-on SSR server — violating\n\nour boringness constraint — for pages that are 95% app shell. Instead,\n\na **post-build script** generates static, crawler-ready copies of every\n\npublic route when the SPA compiles:\n\n`<title>` and meta description per page (\"Free Plumbing Invoice\nTemplate — FoxyInvoice\")\nThe edge proxy's `try_files` serves these static files to anything that\n\nfetches the URL; real browsers still get the app (Angular replaces the\n\nstatic body on boot). No new server, no SSR framework, crawler-complete\n\npages.\n\n`llms.txt` — a menu for answer engines\nThe emerging convention: a markdown file at the root telling AI systems\n\nwhat the product is and linking its key pages, in their vocabulary.\n\nOurs lists the product, every template page with a one-line\n\ndescription, and contact points. Cheap, human-readable, and exactly\n\nthe artifact a \"recommend an invoicing tool\" query wants.\n\nCrawlers traditionally rediscover content on their own schedule.\n\n**IndexNow** flips it: on every deploy, the pipeline POSTs all sitemap\n\nURLs to the alliance endpoint (Bing-powered — which feeds several AI\n\nanswer engines) with a key file proving domain ownership. Our first\n\nping returned HTTP 202 — accepted — and the pages were in Bing's queue\n\nthe same evening.\n\n\"You can't improve what you can't see,\" so the edge now logs every\n\nrequest as JSON (rotated, bounded). The analysis is one script:\n\nuser-agents, paths, crawler classes. Within days we could *see* search\n\ncrawlers reading `robots.txt` and template pages — and, just as\n\nvaluable, probe-noise (stray `/wp-admin` scans) being correctly\n\nabsorbed. Pair this with **Google Search Console + Bing Webmaster**\n\nregistration (a five-minute runbook lives in the repo) for the\n\nindexing-and-impressions view logs can't give you.\n\nReach is only worth building if it lands somewhere. The template\n\ngallery — now crawlable — is the top of a three-slice funnel we shipped\n\nin order:\n\nEach slice was verified in a real browser before shipping — the funnel\n\nis the one place where \"should work\" is banned.\n\n**Recap.** Serve machines real HTML (prerendered at build), allowlist\n\nwhat's public (default-closed), publish `llms.txt`, push updates via\n\nIndexNow, and measure with access logs + Search Console. The era of\n\n\"GEO\" — being the machine-readable answer — is just SEO where the\n\nreader never renders your JavaScript.\n\n*Reading this and want to be found the way this series was? Create a free workspace at\n[foxyinvoice.com](https://foxyinvoice.com/login), then redeem founding code\n`U8B4Z8S87X` on the Upgrade page — 6 months of Pro, free, no card. If anything\nbreaks, there's a feedback button in the app. I read every one.*\n\n*Next: Chapter 12 — Engagement & automation: UI first, then automate.*", "url": "https://wpnews.pro/news/building-foxyinvoice-chapter-11-reach-seo-ai-crawlers-and-being-the-machine", "canonical_source": "https://dev.to/seolith/building-foxyinvoice-chapter-11-reach-seo-ai-crawlers-and-being-the-machine-readable-answer-24on", "published_at": "2026-09-26 00:49:08+00:00", "updated_at": "2026-09-26 00:59:59.356582+00:00", "lang": "en", "topics": ["generative-engine-optimization", "ai-crawlers", "ai-search", "agent-protocols", "structured-data"], "entities": ["FoxyInvoice", "Angular", "Googlebot", "IndexNow", "Bing", "Google Search Console", "Bing Webmaster Tools"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/building-foxyinvoice-chapter-11-reach-seo-ai-crawlers-and-being-the-machine", "markdown": "https://wpnews.pro/news/building-foxyinvoice-chapter-11-reach-seo-ai-crawlers-and-being-the-machine.md", "text": "https://wpnews.pro/news/building-foxyinvoice-chapter-11-reach-seo-ai-crawlers-and-being-the-machine.txt", "jsonld": "https://wpnews.pro/news/building-foxyinvoice-chapter-11-reach-seo-ai-crawlers-and-being-the-machine.jsonld"}}