robots.txt can't block AI crawlers, but something can A developer's analysis of RFC 9309 shows that robots.txt is a voluntary request, not an access-control mechanism, so AI crawlers like GPTBot, ClaudeBot and OAI-SearchBot can ignore it. The piece notes that OpenAI and Anthropic split their crawlers into training, search and user-initiated agents, and recommends enforcing blocks at the request path via CDN/WAF rules, server configuration or IP-range denials while keeping robots.txt as the stated preference. Your robots.txt file does not block anyone. It is a text file full of polite requests, and the standard behind it says so in plain words. RFC 9309, the IETF specification for the Robots Exclusion Protocol, states that the rules "are not a form of access authorization." That matters more than it used to. Your site is now read by AI crawlers that collect training data, search bots that decide whether you appear in chatbot answers, and fetchers that pull your pages when a user asks a question. If you want a say in any of that, it helps to know what robots.txt actually does, who is listening, and what to add when a request is not enough. RFC 9309 formalized the Robots Exclusion Protocol in 2022. The spec says crawlers are requested to honor the rules in your file. Requested. Nothing puts the file between a crawler and your server: a cooperative crawler fetches robots.txt separately and decides for itself whether to comply, and nothing in the protocol can stop a request. Well-behaved crawlers comply because their operators choose to comply. Cloudflare's documentation puts it directly: robots.txt directives are voluntary, and crawlers can ignore them. OpenAI documents three user agents. GPTBot crawls content that may be used to train its foundation models, and disallowing GPTBot tells OpenAI your content should not be used for that training. OAI-SearchBot is used to surface websites in ChatGPT search results; OpenAI says sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. ChatGPT-User fetches a page when a user asks ChatGPT something; because these requests are user-initiated, OpenAI notes that robots.txt rules may not apply. The settings are independent, and OpenAI says search can take about 24 hours to adjust to a robots.txt update. Anthropic documents a similar split. ClaudeBot collects web content that could contribute to model training, Claude-SearchBot works on search result quality, and Claude-User handles fetches directed by a user's question. Anthropic says its bots honor robots.txt directives, supports the non-standard Crawl-delay extension, and asks you to set the rules on every subdomain you want covered. As of October 30, 2026, Anthropic has also begun recognizing a machine-readable signal, based on IETF draft work, that lets a site express a "no AI training" preference, with the rollout happening over time. A robots.txt rule that names ClaudeBot directly takes priority over that general signal. The practical consequence: "allow AI" and "block AI" are not your only options. You can disallow the training crawlers while allowing the search crawlers, so your pages can still be cited in chatbot answers. That split is the part most site owners miss. If you want certainty instead of cooperation, the block has to happen on the request path itself: a CDN or WAF rule, your origin server, or your app requiring a login. Cloudflare's AI Crawler and Agent access policies are the clearest example. You set an action for each behavior Search, Agent, Training , and the Block action stops matching requests at Cloudflare's edge. Two details are worth knowing. The Training behavior also offers a Disallow action that only writes robots.txt directives, which stay voluntary unless you enable the option that additionally blocks AI training crawlers at the edge. And for new domains, all three behaviors default to Allow, so doing nothing is itself a choice. You can do the same thing without Cloudflare: deny requests by user agent in your server configuration, or block the published IP ranges the labs maintain OpenAI publishes a JSON list per bot . One caveat from Anthropic's documentation: blocking their crawler IPs can impede their ability to read your robots.txt, so they warn it may not work as a persistent opt-out on its own. My advice: keep robots.txt as the stated preference, and treat request-path rules as the enforcement layer, not a replacement for the file. Blocking is only half of the decision. The llms.txt proposal, first published in 2024 and updated to v2 in 2026, suggests a markdown file on your site that gives agents concise, expert-level information plus links to markdown versions of your pages. It is a proposal, not a standard, but it notes that thousands of sites now publish one, and OpenAI, Anthropic and Gemini publish their own. If you want AI assistants to describe your product accurately, this is the constructive counterpart to robots.txt: one file states what stays out, the other helps what comes in get read correctly. This part is advice, not documentation. Ask three questions. Do I want my content in training datasets? Do I want it in AI search answers? Do I want user-triggered fetches to work? Then write one robots.txt group per bot to match your answers. If "no" needs to mean "no," put a real block behind it, whether that is a CDN rule, a server rule or a login wall. If you want to be read well, consider an llms.txt. None of it is permanent; you can change the file tomorrow. robots.txt is where you state your preference, and the standard itself says it is not access authorization. The major AI labs document separate bots for training, search and user-triggered fetches, so you can choose per purpose. Only a control on the request path, a CDN rule, your server configuration or a login, can actually stop a request. And llms.txt exists for the opposite goal: helping agents read what you want them to read. The next step is small. Open your robots.txt, decide what it should say about each bot, and decide whether your "no" needs enforcement behind it. Source pages: RFC 9309: Robots Exclusion Protocol https://www.rfc-editor.org/rfc/rfc9309 , OpenAI crawlers https://developers.openai.com/api/docs/bots , Anthropic's crawler documentation https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler , Cloudflare AI Crawler and Agent access policies https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/ , The llms.txt proposal https://llmstxt.org/ . I write about startups and AI at startupwiki.substack.com https://startupwiki.substack.com . Cover photo: Phil Hearing on Unsplash https://unsplash.com/photos/a-no-entry-sign-is-attached-to-a-gate-yOJg-L0y7lM .