{"slug": "what-actually-got-our-articles-cited-by-chatgpt-and-what-didn-t", "title": "What actually got our articles cited by ChatGPT (and what didn't)", "summary": "A developer found that their site's table of contents had been invisible for months due to a legacy anchor mismatch, and that fix was part of a broader effort to get articles cited by AI search engines like ChatGPT and Perplexity. The engineer discovered that answer engine optimization (AEO) requires ensuring crawlers can fetch pages, fixing robots.txt conflicts, adding llms.txt files, and using fragment-addressable headings with proper IDs. The developer also warns that FAQ schema must contain plain text, not markdown, and recommends a site-level entity graph with sameAs arrays.", "body_md": "A few weeks ago I found out our site had a table of contents that rendered for absolutely nobody.\n\nThe code was there. The CSS was there. It had been shipping to production for months. It just never appeared, because the partial looked for `h2[id]`\n\nand our markdown renderer only ever emitted `<a name=\"...\">`\n\n— a legacy anchor, no `id`\n\non the heading itself. So the selector matched zero elements, every time, on every article. Nobody noticed because nobody was looking at it.\n\nI only found it because I was hunting for something else: why our content wasn't getting picked up by AI search — ChatGPT, Perplexity, Google's AI Overviews. We've got ~1,600 published articles. Plenty of traffic historically. And almost nothing showing up as a *citation* in any of the answer engines.\n\nThat hunt turned into a couple of weeks of work. Here's what actually mattered, what didn't, and the stuff I got wrong on the way.\n\nI spent a day reading AEO checklists (that's \"answer engine optimization\" — SEO for the machines that answer instead of link). Every one of them opens with schema markup. None of them start with the question that actually gates everything: **can the AI crawlers even fetch your pages?**\n\nGo look at your `robots.txt`\n\nright now. The bots that feed the answer engines are `GPTBot`\n\n, `OAI-SearchBot`\n\n, `ClaudeBot`\n\n, `PerplexityBot`\n\n, `Google-Extended`\n\n, `CCBot`\n\n. If any of those are disallowed, everything downstream is theater.\n\nThere's a genuinely nasty gotcha here I didn't know. In the robots protocol, a *named* user-agent group **replaces** the `*`\n\ngroup for that bot — it doesn't merge. So if you did the \"nice\" thing and added `User-agent: GPTBot`\n\n/ `Allow: /`\n\nblocks, congratulations: those bots now ignore all your `*`\n\ndisallows and happily crawl `/reactions`\n\n, `/admin`\n\n, whatever. We had GPTBot burning crawl budget on tracking endpoints for exactly this reason. The fix was to delete all the per-bot groups and keep one `User-agent: *`\n\nthat allows content and blocks only the junk. Everyone, bots included, follows it.\n\nThen add an `llms.txt`\n\n— a plain-text, hand-curated list of your best URLs with a one-line description each. It's the \"here's what's worth reading\" file for models. Cheap to write, and it's becoming a real convention.\n\nThis is the part that clicked for me. Answer engines don't cite pages. They cite *passages*. And a passage a model can grab cleanly is one that:\n\nComparison content punches above its weight here, and I think I finally understand why. The rerankers these systems use are good at contrast and negation — \"X does this, Y does *not*.\" So a section literally titled \"Cursor vs Copilot\" with a table underneath is basically pre-chewed for them.\n\nAnd — back to my dead table of contents — put `id`\n\ns on your headings:\n\n``` php\n<!-- legacy, not enough -->\n<h2><a name=\"vram\">VRAM</a></h2>\n\n<!-- what you want -->\n<h2 id=\"vram\">VRAM</h2>\n```\n\nFragment-addressable sections (`/post#vram`\n\n) are easier for a model to isolate and quote. Fixing our renderer was one line. Fixing the *existing* 1,600 articles meant re-rendering all of them, because the HTML was stored, not generated on the fly. Don't forget that step — I almost did.\n\nYes, emit JSON-LD. But Google killed FAQ rich *results* for most sites back in 2023, so if you're doing FAQ schema for the little dropdown in search results, stop — you won't get it. Do it because the answer engines *parse* it.\n\nOne bug I'd bet money you have if you generate FAQ schema from markdown: your answer text is full of markdown. A `[link](url)`\n\nor `**bold**`\n\nsitting inside `acceptedAnswer.text`\n\nlooks broken to a parser. We were doing exactly this. Strip the syntax before you store it — plain text only.\n\nThe one that I think matters most, though, is the site-level entity graph. An `Organization`\n\n+ `WebSite`\n\nblock, emitted on every page, with a `sameAs`\n\narray pointing at your real social/profile URLs. It's how you tell these systems \"this domain is a coherent entity you can trust as a source.\" If you do one schema thing, do this one.\n\nHere's what I can't tell you: whether any of it worked yet.\n\nOur Google traffic is still down about 24% over the last month. The AEO changes are days old. Crawlers re-fetch on their own clock, and the answer engines update slowly, so the earliest I'd expect to see movement is a few weeks out. Anyone promising you a before/after in 48 hours is selling something.\n\nWhat I *can* tell you is the metric to watch, and it's not Google clicks. It's **cited pages** — how many of your URLs show up as sources inside ChatGPT / Perplexity / AI Overviews. Some tools report this now. That number is the one that responds to this work. Track it every couple of weeks, ignore the daily noise.\n\nSo that's the playbook: let the crawlers in, make your paragraphs stand on their own, describe your entity consistently, and measure the right thing. Most of it is unglamorous. The table of contents that rendered for nobody is a decent metaphor for the whole exercise — half of AEO is just fixing the quiet stuff you assumed already worked.\n\n*I run PromptZone, a community for prompt engineering and AI builders. Everything above came out of doing this to our own corpus.*", "url": "https://wpnews.pro/news/what-actually-got-our-articles-cited-by-chatgpt-and-what-didn-t", "canonical_source": "https://dev.to/adifsgaid/what-actually-got-our-articles-cited-by-chatgpt-and-what-didnt-5906", "published_at": "2026-07-23 16:01:30+00:00", "updated_at": "2026-07-23 16:34:05.388121+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "developer-tools"], "entities": ["ChatGPT", "Perplexity", "Google", "GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot", "Google-Extended"], "alternates": {"html": "https://wpnews.pro/news/what-actually-got-our-articles-cited-by-chatgpt-and-what-didn-t", "markdown": "https://wpnews.pro/news/what-actually-got-our-articles-cited-by-chatgpt-and-what-didn-t.md", "text": "https://wpnews.pro/news/what-actually-got-our-articles-cited-by-chatgpt-and-what-didn-t.txt", "jsonld": "https://wpnews.pro/news/what-actually-got-our-articles-cited-by-chatgpt-and-what-didn-t.jsonld"}}