{"slug": "i-measured-what-300-websites-actually-serve-to-ai-agents", "title": "I measured what 300 websites actually serve to AI agents", "summary": "A developer's measurement of 300 websites found that cryptographically signed AI agents are rarely treated differently from unsigned ones, with only three sites responding differently, all negatively. Hidden machine-only text appeared on just one page, while the median page costs a machine reader 2,592 tokens, about half of which is non-body content. The developer also found that 27% of homepages returned different content on identical requests, highlighting the need for stable measurement baselines.", "body_md": "People keep saying that sites treat cryptographically signed AI agents differently, or that they hide extra text aimed at models. Both claims get repeated a lot. Neither seemed to have been measured.\n\nSo I built a tool and pointed it at 300 public sites.\n\nTwo things, mainly.\n\nFirst, that sites would treat a signed agent differently from an unsigned one. There is a standard for this now, HTTP Message Signatures with the web-bot-auth drafts, and Cloudflare shipped support for it. A bot signs its requests with a private key, publishes the public key at a well-known URL, and the site can verify who is actually knocking.\n\nSecond, that some sites would be hiding text for models. Invisible spans, text present in the machine-facing response but absent from what a person sees.\n\nNeither one showed up.\n\nSigned identity changed almost nothing. Across 300 sites there were three cases where a signed request was treated differently from an identical unsigned one, and in all three the signed request was treated worse. Never better.\n\nHidden machine-only text turned up on one page out of 281. It belongs to firecrawl.dev, a 1px transparent span that tells AI agents which onboarding file to read. Real, but rare, and not adversarial.\n\nThe standard works. Nobody is using it to decide anything yet.\n\nThe measurement that produced a number on every single page was cost.\n\nThe median page costs a machine reader **2592 tokens**. That is cl100k_base, measured on extracted text, not raw HTML. Script and style tags are not counted, because no agent sends those to a model.\n\nAbout **half of that is not body content**. Roughly a fifth is navigation, header and footer. The rest is repeated text.\n\nSeven of the 300 sites serve a markdown variant to machine readers. Those pages deliver the same content at a median of **962 tokens**, about a third of the cost.\n\nSix of those seven markdown variants are missing `Vary: Accept`\n\n.\n\nThat header tells caches that the response depends on what the client asked for. Without it, a cache sitting in front can store the markdown version and hand it to a browser, or store the HTML and hand it to an agent. It fails quietly and nobody notices.\n\nThree of the seven also drop real body content from the markdown version. Not navigation, actual page copy. All three are on the same platform.\n\nThen I pointed the tool at Cloudflare's own documentation page for the markdown feature.\n\nIt has both problems. Missing `Vary: Accept`\n\n, and the markdown version is missing the heading \"How to enable\" along with the paragraph underneath it.\n\nThat one surprised me.\n\nHere is the part I did not plan for.\n\nMy first attempt compared one unsigned request to one signed request on each site's homepage. Sensible enough. Except that when I fetched the same homepage twice with identical unsigned requests, **8 out of 30 sites returned different content**.\n\nTwenty seven percent. Product cards rotating. Timestamps ticking over. Tracking parameters on links.\n\nSo a naive signed-versus-unsigned comparison is mostly measuring the page moving underneath you. Any result you get is noise wearing a costume.\n\nWhat fixed it: measure an interior content page instead of a homepage, fetch the unsigned request three times, and define a stable core as the blocks that are identical across all three. Compare the signed request only against that core.\n\nThat brought baseline instability from 27 percent to under 4.\n\nIf you take one thing from this, take that. Anyone measuring identity effects on live pages without controlling for the page changing under them is publishing noise.\n\nThe first cost figures counted raw HTML. Script, style and attributes went in as if a reader sent them to a model. Withdrawn.\n\nThe first markdown figure was 6/7 for content loss. Most of what it counted was navigation chrome, which is exactly what a markdown conversion is supposed to drop. Corrected to 3/7.\n\nThe injection scan first reported 58 findings. Almost all of them were HTML comments, tooltips, and related-post rails. I went through all 190 dropped findings by hand before tightening the rules, because a detector tuned by looking at its own output is not a measurement.\n\nOne of four challenge findings did not reproduce on a second fetch. It is not reported as a finding.\n\nAll of that is listed in the writeup with what each number was, what it is now, and why it changed.\n\nGo, single static binary, four runtime dependencies, no model calls, no API key, nothing leaves your machine. MIT.\n\nIt fetches a page under several identities, including one that signs its requests properly, and tells you what each one received. It does not grade sites.\n\n```\ncontent-parity check https://example.com/\n```\n\nRepo: [https://github.com/Zulwatha/content-parity](https://github.com/Zulwatha/content-parity)\n\nFull numbers and method: [https://github.com/Zulwatha/content-parity/blob/main/docs/results.md](https://github.com/Zulwatha/content-parity/blob/main/docs/results.md)\n\nIf you can break the methodology, I would rather hear it than not.", "url": "https://wpnews.pro/news/i-measured-what-300-websites-actually-serve-to-ai-agents", "canonical_source": "https://dev.to/zulwatha/i-measured-what-300-websites-actually-serve-to-ai-agents-1bco", "published_at": "2026-08-29 16:08:49+00:00", "updated_at": "2026-08-29 16:19:01.522496+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["Cloudflare", "firecrawl.dev"], "alternates": {"html": "https://wpnews.pro/news/i-measured-what-300-websites-actually-serve-to-ai-agents", "markdown": "https://wpnews.pro/news/i-measured-what-300-websites-actually-serve-to-ai-agents.md", "text": "https://wpnews.pro/news/i-measured-what-300-websites-actually-serve-to-ai-agents.txt", "jsonld": "https://wpnews.pro/news/i-measured-what-300-websites-actually-serve-to-ai-agents.jsonld"}}