{"slug": "microsoft-to-publishers-don-t-block-the-ai-bots", "title": "Microsoft to Publishers: Don't Block the AI Bots", "summary": "Microsoft AI VP Nikhil Kolar told publishers and retailers at AdExchanger's Programmatic AI event in Las Vegas last week to stop blocking AI bots and crawlers from their websites. Kolar said four out of five sites currently block such traffic, which renders their content invisible to AI agents and cuts them off from discovery and demand. Microsoft is promoting its Publisher Content Marketplace as a way for publishers to license data to AI firms, with the company acting as a clearing house and earning revenue from cloud compute costs.", "body_md": "Microsoft has one big piece of advice for publishers and retailers that are cautiously navigating the AI era: Let the bots scrape your sites.\n\nThat’s according to Nikhil Kolar, VP of publisher product at Microsoft AI, who spoke at AdExchanger’s Programmatic AI event in Las Vegas last week.\n\nRather than fighting against the rising tide of AI-based search engines and agentic tools, Kolar said publishers and retailers should actually be creating content that can speak directly to these bots (and thus the LLMs generating new forms of traffic). And site owners should also update their robots.txt files to avoid restrictions on crawlers that will diminish the site’s visibility to AI engines.\n\nCurrently, though, four out of five websites block traffic from AI bots and crawlers, according to Kolar. As a result, “the content or products that you have are not legible to agents.” Which means, he added, that “your business is closed” and “you get no discovery, no recommendations that you’re a part of and no demand.”\n\n**Leaning on licensing**\n\nThere is a clearly self-serving angle for Microsoft in pitching publishers on the idea that they ought to open up their sites to AI crawling. After all, Microsoft has its own AI models and AI chatbot searches, and it needs publisher data to inform their outputs.\n\nBut Microsoft also wants publishers to receive fair value for their data, Kolar said. He pointed to Microsoft’s Publisher Content Marketplace as a sign of good faith.\n\nThe initiative, which was announced in February, facilitates licensing agreements between publishers and AI developers. The initial focus was for publishers to monetize by licensing their content or data to be used by Microsoft Copilot. But Microsoft has since [opened the ecosystem](https://digiday.com/media/qa-nikhil-kolar-vp-microsoft-ai-scales-its-click-to-sign-ai-content-marketplace/) to other AI firms, with Microsoft acting as the clearing house for licensing the publisher data.\n\nAdExchanger also spoke with Kolar after his presentation, and he pointed to a distinction between using publisher data for “training” and using it for “grounding” AI models. Simply put, “training” describes establishing the deep initial data pool that underlies an AI model, which is pulled from the vast swathes of content that are already been published online. But “grounding” refers to pulling data from current, trusted sources through more sophisticated [model context protocol (MCP)](https://www.adexchanger.com/adexplainer/understanding-mcp-the-universal-adapter-for-ai-in-advertising/) connections. Microsoft’s focus with its publisher marketplace is on the latter, Kolar told AdExchanger.\n\nTo that point, the publisher marketplace started with just a small roster of premium publishers.\n\nPeople Inc. was first to join Microsoft’s initiative, having already struck deals with OpenAI and Meta, said Jonathan Roberts, chief innovation officer at People Inc., in a fireside chat with the Chief Marketer Network’s senior director of content and community, Lynne d Johnson, that also took place at Programmatic AI.\n\nMicrosoft’s publisher roster has since grown to eight, Roberts said, and Microsoft’s (quite ambitious) goal is to sign up the entire open web.\n\nThe marketplace licensing model has strong selling points to publishers.\n\nBut Roberts noted that the marketplace is mutually beneficial for publishers and Microsoft. Any time a publisher’s data is used to inform an AI tool’s reasoning, the site gets paid, he said. And all of the computing power behind those real-time AI inferences will run on Microsoft’s Azure cloud tech, so Microsoft collects on cloud compute costs.\n\n“That makes it not a cost for Microsoft,” Roberts said. “This is a business.”\n\n**To block or not?**\n\nHowever, Roberts seemed to break from Kolar’s advice to publishers not to block AI crawlers.\n\nInstead, Roberts advised pubs to begin by blocking everything as a way to determine which bots are necessary, such as search indexers and ad-tech-related crawlers. He said People Inc. currently gives access to 38 crawlers and blocks between 30,000 to 35,000 different crawlers a day, adding up to millions of daily unauthorized attempts to scrape its sites.\n\nPlus, blocking all crawlers gives publishers a stronger starting point from which to negotiate said licensing deals, Roberts said. Although he acknowledged that the vast majority of sites on the web have no such clout, even if they block everything.\n\nBut Roberts said, in response to a question from the Prog AI audience about their apparent disagreement, that he thinks he and Kolar are actually in agreement on bot blocking.\n\nKolar’s advice is more aimed at retail and merchant brand sites, Roberts said, since those shopping sites want AI chatbots to recommend their products. Although, he added, large retailers like Walmart might still want to block bot-scraping by default to prevent competitors from building databases of their products.\n\nThe calculus for traditional pubs like People Inc. is different, Roberts said.\n\n“We’re blocking bots so that we can permission them and so we can have control over who has accesses and how,” he said. “That is not a burn-it-all-down approach. That is, ‘Let’s do it the right way.’”\n\nKolar agrees that publishers are tired of reacting and are eager to take more proactive measures.\n\nThis isn’t the web’s first rodeo, after all.\n\nKolar shared how, in some 430 meetings that the Microsoft AI product team had with publishers last year – including 50 he personally attended – the most consistent through line from publishers was a repeated sense of powerlessness when it comes to changes in online behavior by people or platforms.\n\n“The single thing that I heard most from publishers was, ‘Social happened to us, mobile happened to us, search happened to us. We don’t want AI to happen to us,’” he said.\n\nSo, heading into the agentic AI era, Kolar implored publishers to “do something” instead of being merely responsive to painful change that is forced upon them.\n\n“You have a lot more power than you think you have,” he said.", "url": "https://wpnews.pro/news/microsoft-to-publishers-don-t-block-the-ai-bots", "canonical_source": "https://www.adexchanger.com/publishers/microsoft-to-publishers-dont-block-the-ai-bots/", "published_at": "2026-05-26 06:38:14+00:00", "updated_at": "2026-05-26 07:09:27.280278+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-policy", "ai-agents", "large-language-models"], "entities": ["Microsoft", "Nikhil Kolar", "AdExchanger", "Microsoft AI", "Publisher Content Marketplace"], "alternates": {"html": "https://wpnews.pro/news/microsoft-to-publishers-don-t-block-the-ai-bots", "markdown": "https://wpnews.pro/news/microsoft-to-publishers-don-t-block-the-ai-bots.md", "text": "https://wpnews.pro/news/microsoft-to-publishers-don-t-block-the-ai-bots.txt", "jsonld": "https://wpnews.pro/news/microsoft-to-publishers-don-t-block-the-ai-bots.jsonld"}}