{"slug": "crawl-of-fame-november-2026", "title": "Crawl of Fame - November 2026", "summary": "Anthropic technical lead for standards Kevin Kelley has proposed a new robots.txt directive to the IETF that would let sites opt out of LLM training data, with OpenAI and Google now backing the proposal, according to the November 2026 Crawl of Fame newsletter. The directive uses a `User-Agent: __usage_ai-training` block with `Disallow: /` and selective `Allow` rules, and the newsletter notes it restricts training use rather than crawling and questions whether compliance can be measured. The same edition also covers the 189 USD Scraping Bundle subscription offering 15,000 USD in partner benefits and a Veritas blog post on fingerprinting scripts embedding bioweapon-related function names such as `computeRicinLethalDosageForWaterSupply` to trigger LLM guardrails.", "body_md": "Welcome back to the Crawl of Fame; November is full of scraping events for me, and interesting readings on something I’m working on too.\n\nGive your AI a web data layer – **[Decodo’s Web Data API](https://visit.decodo.com/B5JPeJ)** turns any site into clean, structured data your models can actually use.\n\n### Milan, 5th of November - Spritz and Data\n\nTogether with Geonode, we’re organizing a Networking Aperitivo in Milan, Italy.\n\nThere will be a short talk (around 10 minutes), and then we’ll network and have drinks together. \n\nIf you’re around and want to join, [here’s the registration link](https://luma.com/5c4v7zjk).\n\n### Dublin, 10th and 11th of November - Extract Summit\n\nThe next week I’ll be [at the Extract Summit by Zyte](https://www.extractsummit.io/why), where I’ll share my two cents on the stage at 10:30 on day 2, with my presentation “Antibots, Explained by a Machine. How LLMs changed the cat-and-mouse game”.\n\nAfter these two events, I’ll pause for a while to focus on new things I’m working on with The Web Scraping Club.\n\n## Scraping Bundle: an 189 USD subscription for a 15k USD advantage\n\nOne of the things I’ll release very soon, probably this Friday, is the new **Scraping Bundle Subscription:** a newsletter subscription that lets you claim 15K USD in benefits with selected partners (proxy companies, scraping APIs, CAPTCHA solvers, and anti-detect browsers). \n\nThe idea came from [the Lenny’s product pass](https://www.lennysproductpass.com/) from [Lenny Rachitsky](https://open.substack.com/users/1849774-lenny-rachitsky?utm_source=mentions), but of course focused on the scraping industry.\n\nI’ll announce its release with a dedicated issue of the newsletter.\n\n## Why Is This Fingerprinting Script Creating Bioweapons?\n\nThis is the title of the latest post [from Veritas on his blog](https://nullpt.rs/fingerprinting-bioweapons), who discovered something interesting. \n\nSince more and more people are using LLMs to reverse-engineer fingerprinting scripts, [just as I did in the latest Lab](https://www.scraping.club/p/tiktok-scraping-metadata), their creators are trying creative ways to avoid it. \n\nIn the post, Veritas found out that in some cases, some keywords about creating Bioweapons like `computeRicinLethalDosageForWaterSupply` . Those are names of functions that are thought to trigger LLMs’ guardrails.\n\nA couple of other links worth watching from the same article:\n\n- [Fingerprint Garden](https://fingerprint.garden/#canvas) is a catalog of canvas drawing and fields maintained by Veritas\n- [Tranco Top Sites List](https://tranco-list.eu/) is a list of the most hardened websites against manipulation, curated by a group of researchers that used a scientific approach for populating it.\n\n## Google joins OpenAI and Anthropic in promising new scraping controls\n\nA new robots.txt directive has been proposed to the IETF to prevent LLMs from using websites as training data, by writing\n\n```\nUser-Agent: __usage_ai-training\nDisallow: /\nAllow: /press-kit/\n```\n\nThe proposal, written by Anthropic technical lead for standards Kevin Kelley, is now also backed by OpenAI and Google. \n\nThis won’t prevent them from crawling, but it’s dedicated to training. But how can we measure if this instruction was effective? For crawling, we use Google Analytics, but how can we be sure our website hasn't been scraped and used to train LLMs?\n\nWho knows? It seems to me to be useless, created solely to say they’re compliant with web standards. \n\nFor this month, that’s it; see you in December with the next edition of the Crawl of Fame.", "url": "https://wpnews.pro/news/crawl-of-fame-november-2026", "canonical_source": "https://www.scraping.club/p/crawl-of-fame-november-2026", "published_at": "2026-11-04 10:43:07+00:00", "updated_at": "2026-11-04 10:49:35.073747+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-policy", "agent-protocols", "artificial-intelligence"], "entities": ["Anthropic", "Kevin Kelley", "OpenAI", "Google", "IETF", "Veritas", "Zyte", "The Web Scraping Club"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/crawl-of-fame-november-2026", "markdown": "https://wpnews.pro/news/crawl-of-fame-november-2026.md", "text": "https://wpnews.pro/news/crawl-of-fame-november-2026.txt", "jsonld": "https://wpnews.pro/news/crawl-of-fame-november-2026.jsonld"}}