{"slug": "microsoft-clarity-now-flags-bots-that-ignore-robots-txt", "title": "Microsoft Clarity Now Flags Bots That Ignore Robots.txt", "summary": "Microsoft Clarity now flags bot requests that violate a website's robots.txt rules, displaying them as a percentage of total bot activity in its Bot Analytics dashboard. The feature, which must be enabled by project admins for sites using supported CDNs, helps site owners monitor AI crawlers that ignore disallowed paths. This free tool addresses concerns about AI crawlers draining server resources and skewing analytics.", "body_md": "Microsoft Clarity now surfaces bot requests that go against a website’s URL rules in the tool’s Bot Analytics dashboard, the company announced in a [blog post](https://clarity.microsoft.com/blog/robots-txt-violations-in-bot-analytics/).\n\nClarity will calculate and display these requests as a percentage of total bot activity over a given time frame and add to existing AI Visibility tools in the dashboard, which in May began showing [grounding queries behind AI citations](https://www.searchenginejournal.com/microsoft-clarity-now-shows-grounding-queries-behind-ai-citations/575279/).\n\n## What The Violations View Shows\n\nWhen a bot makes a request to a website connected to Clarity, the tool now checks that request against the site’s robots.txt directives to determine if the path was disallowed.\n\nDisallowed bot requests are then calculated and displayed as a percentage of total bot activity over a given time frame.\n\nClarity allows site owners to filter bot requests shown by bot operator, bot name, request activity type, requested URLs and paths, to compare and contrast patterns in crawlers that are known to follow rules with those that don’t.\n\nThis is done by navigating to a side-by-side view comparing crawlers that are generally considered compliant with those showing violations.\n\n## How To Turn It On\n\nThe feature doesn’t activate automatically for all sites and must be enabled by a site’s project admin in the AI Visibility section of Project Settings, specifically for sites using a supported CDN.\n\n[Supported CDNs include](https://clarity.microsoft.com/blog/updates-to-bot-activity-in-clarity/) Fastly, Amazon CloudFront, Cloudflare, Azure Front Door and Akamai. WordPress sites using the latest Microsoft Clarity plugin are also supported.\n\n## Why This Matters\n\nWith the concerns around AI crawlers chewing through server resources and [skewing analytics](https://www.searchenginejournal.com/ai-crawlers-draining-site-resources/543011/), being able to see this activity matters.\n\nAnd since [Clarity is free](https://learn.microsoft.com/en-us/clarity/faq), it’s a no-cost way to keep an eye on whether crawlers honor those rules. It only tells you that the requests happened, not why.\n\nThis data only covers requests that reached paths a site’s robots.txt disallows. Robots.txt is advisory, not something that blocks anything, so Clarity is recording requests that got through rather than ones it stopped.\n\nThe move also acknowledges that manually parsing server logs for bot requests and manually testing URLs against robots.txt to identify disallowed requests is not scalable, with Clarity now automatically counting the number of requests from crawlers that breach a site’s rules.\n\n## Looking Ahead\n\nWebsites now have more accurate, automated ways to assess how well robots.txt rules are being followed.\n\nThe big question is whether making this behavior easier to measure will change how crawlers behave or if it just helps site owners keep a clearer record of what’s happening.", "url": "https://wpnews.pro/news/microsoft-clarity-now-flags-bots-that-ignore-robots-txt", "canonical_source": "https://www.searchenginejournal.com/microsoft-clarity-now-flags-bots-that-ignore-robots-txt/580446/", "published_at": "2026-06-24 20:13:45+00:00", "updated_at": "2026-06-24 20:25:25.920751+00:00", "lang": "en", "topics": ["ai-tools", "ai-ethics", "ai-infrastructure"], "entities": ["Microsoft Clarity", "Fastly", "Amazon CloudFront", "Cloudflare", "Azure Front Door", "Akamai", "WordPress"], "alternates": {"html": "https://wpnews.pro/news/microsoft-clarity-now-flags-bots-that-ignore-robots-txt", "markdown": "https://wpnews.pro/news/microsoft-clarity-now-flags-bots-that-ignore-robots-txt.md", "text": "https://wpnews.pro/news/microsoft-clarity-now-flags-bots-that-ignore-robots-txt.txt", "jsonld": "https://wpnews.pro/news/microsoft-clarity-now-flags-bots-that-ignore-robots-txt.jsonld"}}