{"slug": "making-my-blog-readable-by-ai-agents", "title": "Making My Blog Readable by AI Agents", "summary": "Developer Jake Marsh added machine-readable markdown versions of every post on his static blog Deallocated Objects, serving them either at a .md URL or through Accept-header content negotiation in nginx, so AI agents such as Claude, ChatGPT and Perplexity can read the site without wading through HTML. The build also generates an /llms.txt index, an /llms-full.txt file of all posts (about 225 KB), a JSON Feed with a _markdown_url key per item, and JSON-LD BlogPosting and breadcrumb structured data, while robots.txt explicitly names GPTBot, ClaudeBot, PerplexityBot, Google-Extended and about a dozen other crawlers as allowed. Markdown responses carry a canonical Link header and X-Robots-Tag: noindex to avoid duplicate-content treatment, and an IndexNow key pings Bing when posts change.", "body_md": "# Making My Blog Readable By AI Agents\n\n[Jake Marsh](https://deallocatedobjects.com/about)\n\nA lot of the people reading this blog now aren’t people. They’re agents: Claude, ChatGPT, Perplexity and friends, fetching a page because somebody asked them a question. This week I spent an evening making Deallocated Objects nice to read for them (and for search engines while I was in there).\n\nThe blog is a static site served by nginx, so everything here is either a file generated at build time or a few lines of nginx config. No framework required.\n\n## Markdown For Every Post[#](#markdown-for-every-post)\n\nHTML is a lot of noise for an agent to wade through. Every post now also exists as plain markdown. Add `.md` to any post URL:\n\nEach file starts with a little YAML frontmatter (title, canonical URL, author, dates, a short description), then the post. The older posts were written in markdown to begin with, so the build just cleans those up and points their images at the self-hosted copies.\n\nThe nicer trick is content negotiation. If a client asks for markdown in its `Accept` header, it gets markdown from the *normal* URL, no `.md` needed. In nginx that’s a `map` plus one line per location:\n\nTry it:\n\nTwo details matter here. Every response sends `Vary: Accept`, so a cache in the middle never hands HTML to an agent or markdown to a browser. And the `.md` responses send a `Link: <...>; rel=\"canonical\"` header pointing at the HTML page, plus `X-Robots-Tag: noindex`, so Google doesn’t treat them as duplicate copies of every post.\n\n## llms.txt[#](#llms-txt)\n\n[`/llms.txt`](https://deallocatedobjects.com/llms.txt) is a plain markdown file at the root of the site that tells an agent what it’s looking at. Mine has a one-line summary of the blog, how to get markdown, where the feeds are, how I’d like posts cited, and a list of every post with a link to its markdown and a one-line summary.\n\nThere’s also [`/llms-full.txt`](https://deallocatedobjects.com/llms-full.txt), which is the full text of every post in one file (about 225 KB), and `/index.md`, which is the homepage as markdown. Every response also carries a `Link` header pointing at `llms.txt`, so an agent can find it without guessing.\n\n## robots.txt[#](#robots-txt)\n\nSilence in `robots.txt` technically means “allowed”, but I’d rather be explicit. Mine allows everyone, then names the big ones explicitly (`GPTBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended` and about a dozen more) and adds a `Content-Signal` line:\n\nThat’s the right call for a blog. I *want* to be cited. A product site might choose differently.\n\n## Feeds[#](#feeds)\n\nThe old Atom feed at `/feed` still works, and every entry now has a summary and a `text/markdown` alternate link. I also added a [JSON Feed](https://deallocatedobjects.com/feed.json), where each item carries a `_markdown_url` (JSON Feed allows custom keys that start with an underscore).\n\n## Meta Tags[#](#meta-tags)\n\nThe old site had almost none of these, so every page got the full set:\n\n- a real meta description, a canonical URL, a robots tag and `lang=\"en\"`\n- Open Graph and Twitter card tags, including `article:published_time` and`modified_time`\n- JSON-LD structured data: `BlogPosting` and a breadcrumb on every post, with`WebSite` ,`Blog` and a`Person` for me on the index pages\n\nThe descriptions were sneakier than I expected. The old database stored a pre-made description for each post, but it had run the paragraphs together (“…your app.The RevealWhat do I mean…”) and a few contained raw `<img>` HTML. Now the build makes its summaries from the rendered post text instead.\n\nTwo more fixes from the old 2014 markup. The viewport tag had `maximum-scale=1`, which stops people from pinch-zooming on their phones, so that’s gone. And there was no `og:image` at all, so when I shared a post, X and Threads grabbed the first image on the page, which happens to be the Little Bites of Cocoa banner at the top. Every post has a proper 1200x630 preview card now.\n\n## IndexNow[#](#indexnow)\n\nLast, an [IndexNow](https://www.indexnow.org) key and a script that pings Bing and friends whenever a post is published. Google doesn’t participate, but Bing’s index feeds a lot of AI search tools, so it’s worth a 30-line script.\n\nNone of this took long, and most of it is just files. If you have a blog, `llms.txt` and `.md` versions of your posts are a nice place to start. Here’s this post:\n\nThe rest of how this site is built is in the [colophon](https://deallocatedobjects.com/colophon).", "url": "https://wpnews.pro/news/making-my-blog-readable-by-ai-agents", "canonical_source": "https://deallocatedobjects.com/posts/making-my-blog-readable-by-ai-agents", "published_at": "2026-10-24 21:21:33+00:00", "updated_at": "2026-10-24 21:42:29.342276+00:00", "lang": "en", "topics": ["ai-crawlers", "agent-protocols", "structured-data", "generative-engine-optimization", "ai-search"], "entities": ["Jake Marsh", "Deallocated Objects", "Claude", "ChatGPT", "Perplexity", "GPTBot", "ClaudeBot", "IndexNow"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/making-my-blog-readable-by-ai-agents", "markdown": "https://wpnews.pro/news/making-my-blog-readable-by-ai-agents.md", "text": "https://wpnews.pro/news/making-my-blog-readable-by-ai-agents.txt", "jsonld": "https://wpnews.pro/news/making-my-blog-readable-by-ai-agents.jsonld"}}