A few weeks ago I found out our site had a table of contents that rendered for absolutely nobody.
The code was there. The CSS was there. It had been shipping to production for months. It just never appeared, because the partial looked for h2[id]
and our markdown renderer only ever emitted <a name="...">
— a legacy anchor, no id
on the heading itself. So the selector matched zero elements, every time, on every article. Nobody noticed because nobody was looking at it.
I only found it because I was hunting for something else: why our content wasn't getting picked up by AI search — ChatGPT, Perplexity, Google's AI Overviews. We've got ~1,600 published articles. Plenty of traffic historically. And almost nothing showing up as a citation in any of the answer engines.
That hunt turned into a couple of weeks of work. Here's what actually mattered, what didn't, and the stuff I got wrong on the way.
I spent a day reading AEO checklists (that's "answer engine optimization" — SEO for the machines that answer instead of link). Every one of them opens with schema markup. None of them start with the question that actually gates everything: can the AI crawlers even fetch your pages?
Go look at your robots.txt
right now. The bots that feed the answer engines are GPTBot
, OAI-SearchBot
, ClaudeBot
, PerplexityBot
, Google-Extended
, CCBot
. If any of those are disallowed, everything downstream is theater.
There's a genuinely nasty gotcha here I didn't know. In the robots protocol, a named user-agent group replaces the *
group for that bot — it doesn't merge. So if you did the "nice" thing and added User-agent: GPTBot
/ Allow: /
blocks, congratulations: those bots now ignore all your *
disallows and happily crawl /reactions
, /admin
, whatever. We had GPTBot burning crawl budget on tracking endpoints for exactly this reason. The fix was to delete all the per-bot groups and keep one User-agent: *
that allows content and blocks only the junk. Everyone, bots included, follows it.
Then add an llms.txt
— a plain-text, hand-curated list of your best URLs with a one-line description each. It's the "here's what's worth reading" file for models. Cheap to write, and it's becoming a real convention.
This is the part that clicked for me. Answer engines don't cite pages. They cite passages. And a passage a model can grab cleanly is one that:
Comparison content punches above its weight here, and I think I finally understand why. The rerankers these systems use are good at contrast and negation — "X does this, Y does not." So a section literally titled "Cursor vs Copilot" with a table underneath is basically pre-chewed for them.
And — back to my dead table of contents — put id
s on your headings:
<!-- legacy, not enough -->
<h2><a name="vram">VRAM</a></h2>
<!-- what you want -->
<h2 id="vram">VRAM</h2>
Fragment-addressable sections (/post#vram
) are easier for a model to isolate and quote. Fixing our renderer was one line. Fixing the existing 1,600 articles meant re-rendering all of them, because the HTML was stored, not generated on the fly. Don't forget that step — I almost did.
Yes, emit JSON-LD. But Google killed FAQ rich results for most sites back in 2023, so if you're doing FAQ schema for the little dropdown in search results, stop — you won't get it. Do it because the answer engines parse it.
One bug I'd bet money you have if you generate FAQ schema from markdown: your answer text is full of markdown. A [link](url)
or **bold**
sitting inside acceptedAnswer.text
looks broken to a parser. We were doing exactly this. Strip the syntax before you store it — plain text only.
The one that I think matters most, though, is the site-level entity graph. An Organization
WebSite
block, emitted on every page, with a sameAs
array pointing at your real social/profile URLs. It's how you tell these systems "this domain is a coherent entity you can trust as a source." If you do one schema thing, do this one.
Here's what I can't tell you: whether any of it worked yet.
Our Google traffic is still down about 24% over the last month. The AEO changes are days old. Crawlers re-fetch on their own clock, and the answer engines update slowly, so the earliest I'd expect to see movement is a few weeks out. Anyone promising you a before/after in 48 hours is selling something.
What I can tell you is the metric to watch, and it's not Google clicks. It's cited pages — how many of your URLs show up as sources inside ChatGPT / Perplexity / AI Overviews. Some tools report this now. That number is the one that responds to this work. Track it every couple of weeks, ignore the daily noise.
So that's the playbook: let the crawlers in, make your paragraphs stand on their own, describe your entity consistently, and measure the right thing. Most of it is unglamorous. The table of contents that rendered for nobody is a decent metaphor for the whole exercise — half of AEO is just fixing the quiet stuff you assumed already worked.
I run PromptZone, a community for prompt engineering and AI builders. Everything above came out of doing this to our own corpus.