Welcome back to the Crawl of Fame; November is full of scraping events for me, and interesting readings on something I’m working on too.
Give your AI a web data layer – Decodo’s Web Data API turns any site into clean, structured data your models can actually use.
Milan, 5th of November - Spritz and Data
Together with Geonode, we’re organizing a Networking Aperitivo in Milan, Italy.
There will be a short talk (around 10 minutes), and then we’ll network and have drinks together.
If you’re around and want to join, here’s the registration link.
Dublin, 10th and 11th of November - Extract Summit
The next week I’ll be at the Extract Summit by Zyte, where I’ll share my two cents on the stage at 10:30 on day 2, with my presentation “Antibots, Explained by a Machine. How LLMs changed the cat-and-mouse game”.
After these two events, I’ll for a while to focus on new things I’m working on with The Web Scraping Club.
Scraping Bundle: an 189 USD subscription for a 15k USD advantage #
One of the things I’ll release very soon, probably this Friday, is the new Scraping Bundle Subscription: a newsletter subscription that lets you claim 15K USD in benefits with selected partners (proxy companies, scraping APIs, CAPTCHA solvers, and anti-detect browsers).
The idea came from the Lenny’s product pass from Lenny Rachitsky, but of course focused on the scraping industry.
I’ll announce its release with a dedicated issue of the newsletter.
Why Is This Fingerprinting Script Creating Bioweapons? #
This is the title of the latest post from Veritas on his blog, who discovered something interesting.
Since more and more people are using LLMs to reverse-engineer fingerprinting scripts, just as I did in the latest Lab, their creators are trying creative ways to avoid it.
In the post, Veritas found out that in some cases, some keywords about creating Bioweapons like computeRicinLethalDosageForWaterSupply . Those are names of functions that are thought to trigger LLMs’ guardrails.
A couple of other links worth watching from the same article:
- Fingerprint Garden is a catalog of canvas drawing and fields maintained by Veritas
- Tranco Top Sites List is a list of the most hardened websites against manipulation, curated by a group of researchers that used a scientific approach for populating it.
Google joins OpenAI and Anthropic in promising new scraping controls #
A new robots.txt directive has been proposed to the IETF to prevent LLMs from using websites as training data, by writing
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
The proposal, written by Anthropic technical lead for standards Kevin Kelley, is now also backed by OpenAI and Google.
This won’t prevent them from crawling, but it’s dedicated to training. But how can we measure if this instruction was effective? For crawling, we use Google Analytics, but how can we be sure our website hasn't been scraped and used to train LLMs?
Who knows? It seems to me to be useless, created solely to say they’re compliant with web standards.
For this month, that’s it; see you in December with the next edition of the Crawl of Fame.