Crawl of Fame - November 2026

Anthropic technical lead for standards Kevin Kelley has proposed a new robots.txt directive to the IETF that would let sites opt out of LLM training data, with OpenAI and Google now backing the proposal, according to the November 2026 Crawl of Fame newsletter. The directive uses a `User-Agent: __usage_ai-training` block with `Disallow: /` and selective `Allow` rules, and the newsletter notes it restricts training use rather than crawling and questions whether compliance can be measured. The same edition also covers the 189 USD Scraping Bundle subscription offering 15,000 USD in partner benefits and a Veritas blog post on fingerprinting scripts embedding bioweapon-related function names such as `computeRicinLethalDosageForWaterSupply` to trigger LLM guardrails.

Welcome back to the Crawl of Fame; November is full of scraping events for me, and interesting readings on something I’m working on too. Give your AI a web data layer – Decodo’s Web Data API https://visit.decodo.com/B5JPeJ turns any site into clean, structured data your models can actually use. Milan, 5th of November - Spritz and Data Together with Geonode, we’re organizing a Networking Aperitivo in Milan, Italy. There will be a short talk around 10 minutes , and then we’ll network and have drinks together. If you’re around and want to join, here’s the registration link https://luma.com/5c4v7zjk . Dublin, 10th and 11th of November - Extract Summit The next week I’ll be at the Extract Summit by Zyte https://www.extractsummit.io/why , where I’ll share my two cents on the stage at 10:30 on day 2, with my presentation “Antibots, Explained by a Machine. How LLMs changed the cat-and-mouse game”. After these two events, I’ll pause for a while to focus on new things I’m working on with The Web Scraping Club. Scraping Bundle: an 189 USD subscription for a 15k USD advantage One of the things I’ll release very soon, probably this Friday, is the new Scraping Bundle Subscription: a newsletter subscription that lets you claim 15K USD in benefits with selected partners proxy companies, scraping APIs, CAPTCHA solvers, and anti-detect browsers . The idea came from the Lenny’s product pass https://www.lennysproductpass.com/ from Lenny Rachitsky https://open.substack.com/users/1849774-lenny-rachitsky?utm source=mentions , but of course focused on the scraping industry. I’ll announce its release with a dedicated issue of the newsletter. Why Is This Fingerprinting Script Creating Bioweapons? This is the title of the latest post from Veritas on his blog https://nullpt.rs/fingerprinting-bioweapons , who discovered something interesting. Since more and more people are using LLMs to reverse-engineer fingerprinting scripts, just as I did in the latest Lab https://www.scraping.club/p/tiktok-scraping-metadata , their creators are trying creative ways to avoid it. In the post, Veritas found out that in some cases, some keywords about creating Bioweapons like computeRicinLethalDosageForWaterSupply . Those are names of functions that are thought to trigger LLMs’ guardrails. A couple of other links worth watching from the same article: - Fingerprint Garden https://fingerprint.garden/ canvas is a catalog of canvas drawing and fields maintained by Veritas - Tranco Top Sites List https://tranco-list.eu/ is a list of the most hardened websites against manipulation, curated by a group of researchers that used a scientific approach for populating it. Google joins OpenAI and Anthropic in promising new scraping controls A new robots.txt directive has been proposed to the IETF to prevent LLMs from using websites as training data, by writing User-Agent: usage ai-training Disallow: / Allow: /press-kit/ The proposal, written by Anthropic technical lead for standards Kevin Kelley, is now also backed by OpenAI and Google. This won’t prevent them from crawling, but it’s dedicated to training. But how can we measure if this instruction was effective? For crawling, we use Google Analytics, but how can we be sure our website hasn't been scraped and used to train LLMs? Who knows? It seems to me to be useless, created solely to say they’re compliant with web standards. For this month, that’s it; see you in December with the next edition of the Crawl of Fame.