{"slug": "i-almost-started-a-medium-column-about-getting-cited-by-llms-then-i-read-medium", "title": "I almost started a Medium column about getting cited by LLMs. Then I read Medium's robots.txt", "summary": "A developer's analysis of Medium's robots.txt found the platform blocks eight AI crawlers — including GPTBot, ClaudeBot, Applebot-Extended, meta-externalagent, Bytespider and Amazonbot — while leaving search-grounded retrieval agents such as Googlebot, Bingbot, PerplexityBot, OAI-SearchBot and CCBot unrestricted. The same-day check across dev.to, Hashnode, Substack and Hacker News found no AI crawler rules, while Reddit blocks all crawlers and instead licenses its archive directly to Google and OpenAI. The developer noted the blocked group's allow-list covers only Medium's commercial pages, and that leaving Google-Extended and CCBot open keeps training paths via Google and Common Crawl available.", "body_md": "I was about to start a monthly column about AI search visibility — how to get your pages cited by ChatGPT, Perplexity, Google's AI Mode. Medium was the obvious venue. Big audience, publications that distribute for you, no infrastructure to run.\n\nBefore writing the first paragraph I ran the check I run on every client site:\n\n```\ncurl -s https://medium.com/robots.txt\n```\n\nMedium disallows the crawlers I was writing about.\n\nChecked 2026-09-17, HTTP 200. Alongside the usual `*` rules there's a second group:\n\n```\nUser-Agent: Amazonbot\nUser-Agent: Applebot-Extended\nUser-Agent: Bytespider\nUser-Agent: ClaudeBot\nUser-Agent: FacebookBot\nUser-Agent: GoogleOther\nUser-Agent: GPTBot\nUser-Agent: meta-externalagent\nDisallow: /\nAllow: /about\nAllow: /business\nAllow: /earn\nAllow: /gift\nAllow: /membership\nAllow: /partner-program\nAllow: /verified-authors\n```\n\nEight user-agents, `Disallow: /`, and an allow-list containing exactly the pages that sell Medium. Your post is not on that list.\n\nThat's a defensible business decision — Medium licenses its archive and doesn't want it taken for free. It's just the opposite of what I needed from the platform.\n\nThis is where most takes on this get sloppy, so it's worth being precise. \"AI crawler\" covers two jobs, and Medium only closes one of them.\n\n**Training and bulk collection — closed.** `GPTBot` (OpenAI), `ClaudeBot` (Anthropic), `Applebot-Extended` (Apple Intelligence), `meta-externalagent` (Meta), `Bytespider` (ByteDance), `Amazonbot`.\n\n**Search-grounded retrieval — open.** These are not in the blocked group, so the `*` rules apply and they're allowed:\n\n| user-agent | what it feeds | \n|---|---|\n| `Googlebot` | Google's index → AI Overviews, AI Mode | \n| `Bingbot` | Bing's index → Copilot | \n| `PerplexityBot` | Perplexity's own index | \n| `OAI-SearchBot` | ChatGPT's search index | \n| `ChatGPT-User` | user-triggered fetch when ChatGPT opens a link | \n| `Google-Extended` | Gemini / Vertex training use of Google-crawled pages | \n| `CCBot` | Common Crawl | \n\nSo a Medium post can absolutely turn up as a cited source in Perplexity or an AI Overview. It reaches those engines through a search index, and those indexes are open.\n\nTwo of those rows deserve a second look.\n\n`Google-Extended` isn't blocked. It doesn't crawl anything itself — it's the token that controls whether content Google already crawled may be used for Gemini training. Left unblocked, that path stays open while OpenAI's and Anthropic's equivalents are shut.\n\n`CCBot` isn't blocked either. Common Crawl is a public archive that a lot of training corpora are built from. Blocking the labs' own crawlers while leaving the archive they can download open is a narrower measure than it first looks.\n\nI don't read this as a loophole anyone left on purpose. I read it as the normal state of a robots.txt: a list of names, maintained by hand, always slightly behind a user-agent landscape that adds a new one every few months.\n\nSame command, same day:\n\n| platform | AI crawler rules in the `*` group | \n|---|---|\n| `medium.com` | separate group blocking 8 AI user-agents | \n| `dev.to` | none — only `/*/actions_panel*` disallowed | \n| `hashnode.com` | none — `Allow: /` | \n| `substack.com` | none — only `/action/` disallowed | \n| `news.ycombinator.com` | none — open, `Crawl-delay: 30` | \n| `reddit.com` | `User-agent: *` +`Disallow: /` — everything blocked | \n\nReddit is the interesting one. It doesn't single out AI crawlers; it disallows all of them, then licenses the archive to Google and OpenAI directly. Reddit threads do reach those two models, but through a contract, not through a crawl. If you were counting a Reddit comment as third-party evidence any model could go verify, that's not what it is.\n\nA model reading your own domain treats what it finds as a self-description. The same sentence on a domain you don't control reads differently — and that difference is the entire reason to publish somewhere other than your own site.\n\nI have a concrete case. A comparison prompt to Gemini described my tool as a *\"Local SEO & AI Indexing\"* product *\"built on top of Scrapeless.\"* There is no such product underneath it. The acronym did the damage: a lot of indexed repositories use GEO and \"geographic\" interchangeably, so a model reading a bare GEO with no nearby context can resolve it to the geographic sense and attach whatever vendor sits in that space.\n\nThe fix was boring. Put the expansion next to the name where a model reads first — hero paragraph, meta description, the `WebSite` and `WebApplication` nodes in the JSON-LD graph. On the built HTML, the entity-disambiguation check went from 1/3 to 3/3 and the citability score from 60 to 62.\n\nBut that fix only works on pages I own. The correction that matters more is the one on a domain that isn't mine — which is why the platform's robots.txt is a prerequisite, not a detail.\n\nBefore you commit a column to a platform, read the platform's robots.txt. It's one command, and it tells you which of the two channels you're buying.\n\nThen be honest about which you needed:\n\nI wanted the second, so this column is on dev.to.\n\nThree caveats worth stating plainly. `robots.txt` is a request, not enforcement — it tells you a platform's stated policy, not what every crawler does. The lists change; mine is a reading from one day, and yours should be your own. And a publication on a custom subdomain can serve a different file than its parent, so check the host you'll actually publish under.\n\nIf you want the same check across the AI user-agents on your own domain, that's the first thing [the CLI I maintain](https://geoready.dev) does — `geo access` against your robots.txt. But you don't need it for this. You need `curl` and two minutes, on the platform you were about to hand your writing to.", "url": "https://wpnews.pro/news/i-almost-started-a-medium-column-about-getting-cited-by-llms-then-i-read-medium", "canonical_source": "https://dev.to/juanauriti/i-almost-started-a-medium-column-about-getting-cited-by-llms-then-i-read-mediums-robotstxt-1ak3", "published_at": "2026-09-17 07:48:21+00:00", "updated_at": "2026-09-17 07:53:33.087599+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "ai-search", "structured-data"], "entities": ["Medium", "OpenAI", "Anthropic", "Google", "Perplexity", "Reddit", "Common Crawl", "ByteDance"], "alternates": {"html": "https://wpnews.pro/news/i-almost-started-a-medium-column-about-getting-cited-by-llms-then-i-read-medium", "markdown": "https://wpnews.pro/news/i-almost-started-a-medium-column-about-getting-cited-by-llms-then-i-read-medium.md", "text": "https://wpnews.pro/news/i-almost-started-a-medium-column-about-getting-cited-by-llms-then-i-read-medium.txt", "jsonld": "https://wpnews.pro/news/i-almost-started-a-medium-column-about-getting-cited-by-llms-then-i-read-medium.jsonld"}}