Making My Blog Readable by AI Agents

Developer Jake Marsh added machine-readable markdown versions of every post on his static blog Deallocated Objects, serving them either at a .md URL or through Accept-header content negotiation in nginx, so AI agents such as Claude, ChatGPT and Perplexity can read the site without wading through HTML. The build also generates an /llms.txt index, an /llms-full.txt file of all posts (about 225 KB), a JSON Feed with a _markdown_url key per item, and JSON-LD BlogPosting and breadcrumb structured data, while robots.txt explicitly names GPTBot, ClaudeBot, PerplexityBot, Google-Extended and about a dozen other crawlers as allowed. Markdown responses carry a canonical Link header and X-Robots-Tag: noindex to avoid duplicate-content treatment, and an IndexNow key pings Bing when posts change.

Making My Blog Readable By AI Agents Jake Marsh https://deallocatedobjects.com/about A lot of the people reading this blog now aren’t people. They’re agents: Claude, ChatGPT, Perplexity and friends, fetching a page because somebody asked them a question. This week I spent an evening making Deallocated Objects nice to read for them and for search engines while I was in there . The blog is a static site served by nginx, so everything here is either a file generated at build time or a few lines of nginx config. No framework required. Markdown For Every Post markdown-for-every-post HTML is a lot of noise for an agent to wade through. Every post now also exists as plain markdown. Add .md to any post URL: Each file starts with a little YAML frontmatter title, canonical URL, author, dates, a short description , then the post. The older posts were written in markdown to begin with, so the build just cleans those up and points their images at the self-hosted copies. The nicer trick is content negotiation. If a client asks for markdown in its Accept header, it gets markdown from the normal URL, no .md needed. In nginx that’s a map plus one line per location: Try it: Two details matter here. Every response sends Vary: Accept , so a cache in the middle never hands HTML to an agent or markdown to a browser. And the .md responses send a Link: <... ; rel="canonical" header pointing at the HTML page, plus X-Robots-Tag: noindex , so Google doesn’t treat them as duplicate copies of every post. llms.txt llms-txt /llms.txt https://deallocatedobjects.com/llms.txt is a plain markdown file at the root of the site that tells an agent what it’s looking at. Mine has a one-line summary of the blog, how to get markdown, where the feeds are, how I’d like posts cited, and a list of every post with a link to its markdown and a one-line summary. There’s also /llms-full.txt https://deallocatedobjects.com/llms-full.txt , which is the full text of every post in one file about 225 KB , and /index.md , which is the homepage as markdown. Every response also carries a Link header pointing at llms.txt , so an agent can find it without guessing. robots.txt robots-txt Silence in robots.txt technically means “allowed”, but I’d rather be explicit. Mine allows everyone, then names the big ones explicitly GPTBot , ClaudeBot , PerplexityBot , Google-Extended and about a dozen more and adds a Content-Signal line: That’s the right call for a blog. I want to be cited. A product site might choose differently. Feeds feeds The old Atom feed at /feed still works, and every entry now has a summary and a text/markdown alternate link. I also added a JSON Feed https://deallocatedobjects.com/feed.json , where each item carries a markdown url JSON Feed allows custom keys that start with an underscore . Meta Tags meta-tags The old site had almost none of these, so every page got the full set: - a real meta description, a canonical URL, a robots tag and lang="en" - Open Graph and Twitter card tags, including article:published time and modified time - JSON-LD structured data: BlogPosting and a breadcrumb on every post, with WebSite , Blog and a Person for me on the index pages The descriptions were sneakier than I expected. The old database stored a pre-made description for each post, but it had run the paragraphs together “…your app.The RevealWhat do I mean…” and a few contained raw <img HTML. Now the build makes its summaries from the rendered post text instead. Two more fixes from the old 2014 markup. The viewport tag had maximum-scale=1 , which stops people from pinch-zooming on their phones, so that’s gone. And there was no og:image at all, so when I shared a post, X and Threads grabbed the first image on the page, which happens to be the Little Bites of Cocoa banner at the top. Every post has a proper 1200x630 preview card now. IndexNow indexnow Last, an IndexNow https://www.indexnow.org key and a script that pings Bing and friends whenever a post is published. Google doesn’t participate, but Bing’s index feeds a lot of AI search tools, so it’s worth a 30-line script. None of this took long, and most of it is just files. If you have a blog, llms.txt and .md versions of your posts are a nice place to start. Here’s this post: The rest of how this site is built is in the colophon https://deallocatedobjects.com/colophon .