Don’t block the bots. Build the gate

Media companies are blocking AI training bots but missing the opportunity to allow retrieval bots that can boost their presence in AI summaries. Publishers should adopt a nuanced strategy that treats bots as part of the audience, prioritizing authority over direct traffic. Allowing certain bots, like retrieval bots, can help maintain mind share and competitive positioning in AI-driven search.

Media companies have gotten pretty good at blocking AI https://www.fastcompany.com/section/artificial-intelligence bots. They have been less good at deciding which ones should be allowed in. Reports say https://pressgazette.co.uk/platforms/eight-in-ten-of-worlds-biggest-news-websites-now-block-ai-training-bots/ most major publications almost universally block crawlers from major AI companies like OpenAI and Anthropic, and many small to midsize publishers mirror that, configuring their robots exclusion protocol settings, or robots.txt, to keep them out. It’s an understandable stance, and it might even be appropriate. But it also crudely compresses a complex issue into a binary: Block or don’t block. And the reasoning is often just as straightforward, pointing to the fact that most AI platforms don’t give back significant human traffic. At the same time, the number of bot crawls can be extreme, high enough that cost becomes a factor. From an ROI standpoint, it’s an easy call. There are exceptions, of course. Google is the obvious complicated one, since AI is increasingly built into Search itself, and blocking Googlebot means opting out of a lot more than AI answers https://www.fastcompany.com/91566646/publishers-cant-control-ai-answers-they-cant-ignore-them-either . Google says AI Overviews and AI Mode are part of Search https://developers.google.com/search/docs/appearance/ai-features , and that site owners control inclusion through the same Search crawling and preview controls they already use. And certainly, if your publication has a licensing deal with a particular platform, you’d let its bots through. But if direct audience or revenue is the only criterion for allowing bots in, that’s too narrow a window to really take advantage of the opportunity AI presents. Ultimately, blocking is just one component of an AI-forward strategy that understands the different types of bots, how they connect your content to audiences, the strategic importance of treating bots as part of the audience https://www.fastcompany.com/91555437/bots-audience-now-changes-everything-media , and how to present content in AI experiences so you stay in control. Let’s start with the truth that different bots do different things. There are several types, but broadly three matter most to publishers: training bots, search bots, and retrieval bots. At least those are the types that are part of the “legitimate” information ecosystem, in that they are usually declared and documented, and generally respect robots.txt preferences. Let’s put unauthorized scrapers aside for the moment. It’s the retrieval bots, which typically have “-user” as part of their name, that publishers should pay close attention to and consider giving the green light. These are the bots that go and get information on the fly when a user types a question into an AI chatbot. Their activity is modest and predictable, usually hitting only a small number of pages, and they can often be rate-limited or served cached content. Cloudflare, in fact, has made it easier to distinguish between uses like search indexing, real-time AI input, and AI training, and its AI Crawl Control tools https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/ now include options to allow, charge, or block specific AI crawlers. Allowing retrieval bots, and potentially a rate-limited number of search bots, can have a meaningful effect on presence in AI summaries. “So what?” is the question many publishers retort—you can’t take AI presence to the bank. Which is true, but it misses the strategic point: If you’re not in the answer, someone else is. That cedes authority on a subject to a competitor, because the more audiences see them cited, the more they’ll develop mind share, and what little referral traffic there is will go to them. Winning in AI means understanding that authority is the prize https://www.fastcompany.com/91543448/ai-search-is-creating-a-new-incentive-system-for-media , not traffic. Winning that authority, however, doesn’t mean winning at all costs. Your most valuable assets—whether they be content, community, or experiences—need to remain yours, not given freely to AI systems. Strategic content in particular requires a careful balance between what you allow AI to see and what readers need to come directly to you to discover. If you closely manage metadata, snippets, access controls, and clear instructions for LLMs, you can help bots understand that valuable material exists without giving it away entirely. For example, a site specialized in the agricultural industry might have an extensive, data-rich report on the effects of pesticides on soybean demand. A snippet might describe exactly what kind of data is within and why the data is relevant to certain types of research while not revealing the data or conclusions. The report itself might make clear what is available, what is restricted, and where authorized users can access the full work. Ideally, the result would be that the existence of the report is known to AI engines, but anyone who wanted to actually see it would need to visit the publisher and engage deeper, with a subscription or at least an email gate. The goal isn’t to hide entirely from AI answer engines. It’s to make them aware of the publisher’s value without letting them reproduce that value. There’s an analogy here with what Palantir cofounder Alex Karp said in a widely shared CNBC interview https://www.cnbc.com/2026/07/01/palantir-karp-open-ai-anthropic-tokens.html , where he advised enterprise AI customers to stop using models from the major AI labs, since it was effectively giving them their “alpha”—the company data that gives them an edge over competitors. This is, in a sense, what media companies have been forced to do from the beginning of the AI revolution, since publishers’ data is largely public and was a big part of what the models originally trained on. Now the creators of those models are effectively competing with publishers by offering answer engines to the public, which keep users at the portal instead of sending them to sites. This is, of course, the foundational idea behind the many lawsuits and licensing deals between the AI companies and the media. To give users the best and most up-to-date information, though, you need to get it at the moment of the query, which is why the market has shifted from training data to retrieval. Rob Kelly has a useful analysis of this shift https://mediaandthemachine.substack.com/p/ai-content-licensing-fewer-deals , showing that training rights are no longer a given in publicly announced AI licensing deals. In his database, only about 4 in 10 public 2026 deals include training rights, a sign that the market is moving from “buy content to build better models” toward “license content to deliver better answers.” The sad truth is that today the vast majority of publishers don’t have the leverage to secure their own licensing deals with OpenAI, et al. But that doesn’t mean there aren’t ways to extract the value of your content in an AI marketplace. It starts with making your content machine-readable, a separate concept from bot blocking. The number of places publishers can market their content to AI experiences is growing, including Factiva and Microsoft’s Publisher Content Marketplace, which Microsoft describes https://about.ads.microsoft.com/en/blog/post/february-2026/building-toward-a-sustainable-content-economy-for-the-agentic-web as a way to support licensed access to premium content while preserving publisher control, independence, and sustainable revenue. Finally, building your own agentic layer that legitimate customers or partners could access via MCP model context protocol is the most forward-looking option. That takes some technical sophistication, often requiring a partner, but it gives the publisher control over the experience. Rather than relying on some third-party bot to scrape and interpret your content, you’ll have done that work already, so authorized external bots will get what your internal bot has already processed, like a relay. That’s the whole idea behind AI media projects like Reuters’ new MCP server https://www.reuters.com/media-center/reuters-launches-model-context-protocol-server-bring-trusted-news-directly-into-2026-07-08/ , which allows customers to search, retrieve, and use the Reuters content they subscribe to inside AI workflows. As I’ve argued before about publisher-built agents and AI-ready archives https://www.fastcompany.com/91442645/ai-chatbots-wont-save-media-what-powers-them-might , once you’ve done the hard work of formatting, ingesting, and processing your archive for AI, the publisher becomes something more like a tool vendor, not just a content supplier. Reuters clearly recognizes that publishers who don’t build some kind of controlled retrieval layer over their archives are letting third-party crawlers set the rules. Eventually, that will lead to negotiating from weakness. I haven’t said much about the bad actors so far—the unauthorized scrapers that hoover up huge parts of what publishers produce and then sell that data in various black and gray markets. Certainly, this is where blocking should be encouraged, and publishers need both reliable tools for doing so as well as broader ecosystem support, such as what Cloudflare has done to better identify bots and potentially monetize their activity. As I wrote earlier this year https://www.fastcompany.com/91504520/publishers-are-finally-getting-serious-about-ai-scraping , unauthorized AI crawling is rampant, and publishers need more than wishful thinking and a robots.txt file to deal with it. But blocking is only the defensive half of the strategy. The answer is not to throw the doors open or nail them shut. It is to build a gate with rules. Let the retrieval bots see enough to establish your authority. Keep the training crawlers and unauthorized scrapers away from anything they have no right to absorb. Use snippets, metadata, paywalls, and rate limits to separate discovery from access. And then do the harder work: Make your archive clean, structured, machine-readable, and available through channels you control. Most publishers are not going to get the giant AI licensing deal. Which is okay. That was never going to be a business plan anyway. The better play is to make your work legible to AI systems without making it free, so when the market does come looking for trusted answers, you’re not begging to be included. You’re already the gate.