Microsoft Clarity now surfaces bot requests that go against a website’s URL rules in the tool’s Bot Analytics dashboard, the company announced in a blog post.
Clarity will calculate and display these requests as a percentage of total bot activity over a given time frame and add to existing AI Visibility tools in the dashboard, which in May began showing grounding queries behind AI citations.
What The Violations View Shows #
When a bot makes a request to a website connected to Clarity, the tool now checks that request against the site’s robots.txt directives to determine if the path was disallowed.
Disallowed bot requests are then calculated and displayed as a percentage of total bot activity over a given time frame.
Clarity allows site owners to filter bot requests shown by bot operator, bot name, request activity type, requested URLs and paths, to compare and contrast patterns in crawlers that are known to follow rules with those that don’t.
This is done by navigating to a side-by-side view comparing crawlers that are generally considered compliant with those showing violations.
How To Turn It On #
The feature doesn’t activate automatically for all sites and must be enabled by a site’s project admin in the AI Visibility section of Project Settings, specifically for sites using a supported CDN.
Supported CDNs include Fastly, Amazon CloudFront, Cloudflare, Azure Front Door and Akamai. WordPress sites using the latest Microsoft Clarity plugin are also supported.
Why This Matters #
With the concerns around AI crawlers chewing through server resources and skewing analytics, being able to see this activity matters.
And since Clarity is free, it’s a no-cost way to keep an eye on whether crawlers honor those rules. It only tells you that the requests happened, not why.
This data only covers requests that reached paths a site’s robots.txt disallows. Robots.txt is advisory, not something that blocks anything, so Clarity is recording requests that got through rather than ones it stopped.
The move also acknowledges that manually parsing server logs for bot requests and manually testing URLs against robots.txt to identify disallowed requests is not scalable, with Clarity now automatically counting the number of requests from crawlers that breach a site’s rules.
Looking Ahead #
Websites now have more accurate, automated ways to assess how well robots.txt rules are being followed.
The big question is whether making this behavior easier to measure will change how crawlers behave or if it just helps site owners keep a clearer record of what’s happening.