The Web Is Sorting Bots by Purpose, Not by Behavior #
Every anti-bot system built in the last fifteen years asks one question: does this traffic look automated? Header order, wire-level fingerprints, mouse movement, session timing. All of it tries to infer something about the client from the shape of the request.
On July 1, 2026, Cloudflare changed the question. Bot traffic now sorts into three categories that have nothing to do with what a request looks like on the wire. Search covers "crawlers that index your content so they can answer questions about it later." Agent covers "automated activity acting in real time on a person's behalf." Training covers "crawlers that take your content to train or fine-tune a model."
Picture three requests arriving at the same server. Same client, same headers, same timing, identical down to the byte. Under this model they can get three different verdicts, decided entirely by what you plan to do with the page after it arrives.
That's not a better detector. It's a different primitive.
The September 15 Deadline Is Narrower Than the Panic
The version of this story going around developer forums is that Cloudflare blocks AI agents across a fifth of the web on September 15.
The changelog says something much smaller. The new defaults apply to "new domains onboarding to Cloudflare." On those domains, "bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed." Everyone already on Cloudflare chooses their own setting, and can do it any time before the date.
So: new domains only, ad-supported pages only, and one of the three categories still gets through by default. That's a real change and it isn't a wall across the internet.
But the narrow scope is the interesting part. Cloudflare didn't publish a policy, it published a default, and defaults are how a policy becomes the norm without anyone voting on it. The number worth watching isn't September 15. It's how many of those domains never touch the setting afterward.
What this means: if you collect data, the useful question stopped being "can I get through this site's protection" and started being "which category does this site think I'm in." Those have different answers and only one of them is something you can test for.
Purpose Can't Be Measured. It Has to Be Declared.
Here's the mechanical problem with sorting traffic by intent: intent isn't in the packet.
A fingerprint can be measured. Mouse movement can be measured. "Why are you fetching this page" can't, so the system needs the client to say it out loud, and it needs a reason to believe the answer. That's the job Web Bot Auth does. The agent signs its requests with a private key, publishes a key directory, and the edge verifies the signature against it, built on RFC 9421 HTTP Message Signatures. Cloudflare's docs set the bar for a verified bot as "honest self-identification."
Now look at what's enforcing this at scale. The spec is draft-meunier-webbotauth-httpsig-protocol-02, last revised August 18, 2026. Intended status: Standards Track. Actual status: an "Active Internet-Draft (individual)" that's "not endorsed by the IETF" and carries "no formal standing in the IETF standards process." It has two authors. One works at Cloudflare, one works at Google.
We think the design is right, and that's worth saying before the criticism lands. Signed identity beats the captcha arms race for anyone who intends to behave. A crawler that identifies itself can be allowed, throttled, or billed on purpose instead of guessed at, and site owners get a control that doesn't take out real visitors as collateral. Set that against a decade of blocking IP ranges and hoping.
What it can't do is stop anybody. A system built on declared purpose only sorts the traffic that declares itself. Everything else falls back to the detection stack that was already there, and that stack keeps getting sharper: Cloudflare's session-scoped behavioral scoring shipped twelve days after the new categories did.
So the two aren't competing. Identity sorts the honest, detection handles everyone else, and the second one is where your traffic lands by default if you never declare anything.
There's No Box for Collecting Public Data
Now the part that should bother anyone running a collection pipeline.
Take the work we've published this year. A timber price index built from public listings. MAP violation detection across six marketplaces, which means fetching the same product page from six different vantage points and comparing what each one says. App store rankings and review sentiment. Ad placement checks run from the country the campaign was actually bought in.
Sort those into Search, Agent, or Training.
Search doesn't fit: Cloudflare's own definition attaches an expectation of "referral traffic or other equitable compensation in return," and a nightly price check sends referrals to nobody. Agent doesn't fit either, because no human is sitting in front of a screen waiting on that fetch. And Training is plainly wrong, since nothing gets absorbed into a model.
This is a taxonomy written to describe the AI traffic that showed up in the last three years, now applied to a business that predates it by a decade. Price intelligence, alternative data, SERP tracking, brand protection, ad verification, compliance monitoring: none of it is new, none of it is agentic, and none of it has a box to check.
Traffic with no box gets sorted by whoever drew the boxes. Usually into the nearest one.
What This Means for Data Teams #
Four things worth doing while the rules are still soft.
Decide which category you'd honestly claim. Not the one that gets you through the door. The one you'd defend in writing if a publisher asked you directly. If the honest answer is "none of these," then you're in the group with the most to lose when the taxonomy hardens, and the most to gain from saying so while there's still an argument to be had.
Read the target's settings, not the headlines. A site's bot policy is now a property of that site, configurable per path, and it changes without notice. Treating "Cloudflare blocks agents in September" as a fact about your pipeline will send you rebuilding things that were never at risk.
Assume the ad-page test spreads. Cloudflare tied its default to pages that display ads, a decent proxy for "this page earns money from human attention." That line will move, and Cloudflare isn't the only one drawing it. AWS WAF shipped AI traffic monetization on June 15, 2026, answering with a 402 and machine-readable payment terms. Akamai took the partnership route with TollBit and Skyfire. Three of the largest edge providers now sell the same control surface, which we looked at from the pricing side when pay-per-crawl launched.
Keep your collection able to work in either lane. The identified lane (sign, declare, get allowed or get billed) and the unidentified one (get judged on behavior, like always) are going to coexist for years. Building as though only one of them exists is the expensive mistake here, in both directions.
The Boxes Are Going to Be Redrawn #
A prediction we're happy to be held to: a fourth category shows up within a year.
The first version of any taxonomy gets written by whoever's under pressure, and the people being classified aren't in the room when it happens. Search, Agent and Training describe what AI companies do to publishers. They don't describe a market research firm, a compliance team, or a retailer checking a competitor's shelf price, and those three have been fetching public pages politely since before "agent" meant anything in this context.
The fight over what that fourth box is called, and who gets to check it, will matter more to the data industry than any anti-bot benchmark published this year. That one's worth showing up for.
Because the fallback is simple enough. If your traffic can't name itself, somebody else will name it for you.