TLDR We pulled verified anti-scraping clauses out of the Terms of Service of 35 AI companies and platforms and 150 top consumer sites, vectorized them as character 4-grams with TF-IDF weights, and linked every cross-company pair at a cosine of 0.35 or higher. The biggest group that came out is the "robot, spider, scraper" template, shared by 17 sites. It's standard-library Python, and on the study's quotes it gives the same similarities as scikit-learn's TfidfVectorizer with a character analyzer.
For the AI Hypocrisy Index we scored how hard each company's Terms ban scraping against how broadly it claims user or public data. Side by side, many of the anti-scraping clauses read like one sentence with a word swapped. Netflix's Terms say "use any robot, spider, scraper or other automated means to access the Netflix service." Perplexity's say "use any robot, spider, crawlers, scraper, or other automatic device, process, software or queries."
We measured the overlap with code, since there are far too many pairs to compare by eye. The input is the study's evidence table, one row per clause. A clause only got into that table when its quote turned up as an exact substring of the fetched document. That check matters for similarity work, because a paraphrase from the classifier would make two companies look more alike, or less alike, than their legal text is.
Each row carries a clause_type whose prefix names the clause. The prefixes AS1 to AS7 mark the seven anti-scraping types (a general ban, an absolute ban, prior written consent, a ban on AI training, liquidated damages, anti-circumvention and API-only access), and the EX types on the extraction side are left out. A quote under 20 characters has only a handful of n-grams and matches too easily, so it's dropped.
import csv
ANTI_SCRAPING = {"AS1", "AS2", "AS3", "AS4", "AS5", "AS6", "AS7"}
def load_clauses(path):
clauses = []
with open(path, encoding="utf-8", newline="") as f:
for row in csv.DictReader(f):
kind = row["clause_type"].split("_")[0]
quote = row["verbatim_quote"].strip()
if kind in ANTI_SCRAPING and len(quote) >= 20:
clauses.append({"company": row["company"], "type": kind, "quote": quote})
return clauses
The expects a CSV with company, clause_type and verbatim_quote columns, and it keeps only rows whose type starts with AS1 to AS7, so label your own clause table that way or edit the set. The IDF weights come from whichever quotes you load, which means the 0.30 and 0.35 cuts below were set on the study's table and are worth re-checking on yours.
Legal wording drifts in ways that word tokens handle badly. "Crawler" and "crawlers" are different words, and so are "automated" and "automatic", but as character 4-grams they share most of their pieces. Lowercasing and collapsing whitespace first keeps line breaks from the source documents out of the n-grams.
TF-IDF does the other half of the job. N-grams that show up in most clauses, such as the pieces of "or" and "any", get a low IDF, so the similarity between two quotes comes from the phrasing they share with few others. Each vector is L2-normalized, which turns cosine similarity into a plain dot product.
import re
from collections import Counter
from math import log, sqrt
def char_ngrams(text, n=4):
text = re.sub(r"\s+", " ", text.lower()).strip()
return [text[i:i + n] for i in range(len(text) - n + 1)]
def tfidf_vectors(docs, n=4):
grams = [char_ngrams(d, n) for d in docs]
df = Counter(g for doc in grams for g in set(doc))
idf = {g: log((len(docs) + 1) / (df[g] + 1)) + 1 for g in df}
vectors = []
for doc in grams:
vec = {g: tf * idf[g] for g, tf in Counter(doc).items()}
norm = sqrt(sum(v * v for v in vec.values())) or 1
vectors.append({g: v / norm for g, v in vec.items()})
return vectors
def cosine(a, b):
if len(a) > len(b):
a, b = b, a
return sum(v * b.get(g, 0) for g, v in a.items())
The smoothed IDF, log((N + 1) / (df + 1)) + 1, is the formula scikit-learn uses by default. On our quotes the dictionary version above and TfidfVectorizer agreed to floating-point precision, so if you're already using scikit-learn (pip install scikit-learn), five lines give the same similarity matrix. They part ways on whitespace. Scikit-learn's character analyzer only collapses runs of two or more whitespace characters, so a lone line break inside a quote stays in its n-grams.
from sklearn.feature_extraction.text import TfidfVectorizer
clauses = load_clauses("evidence.csv")
quotes = [c["quote"] for c in clauses]
X = TfidfVectorizer(analyzer="char", ngram_range=(4, 4)).fit_transform(quotes)
similarity = (X @ X.T).toarray()
Calling toarray() builds an N by N matrix of floats, which comes to about 200 MB at 5,000 quotes. Past a few thousand quotes it's better to keep it sparse and read only the rows you need. To run the pairing step on it, read similarity[i, j] where similar_pairs calls cosine(vectors[i], vectors[j]).
Every quote is compared with every other one, and pairs from the same company are skipped. A company that repeats its own ban in two sections of its Terms doesn't tell you whether it shares a template with anyone else.
def similar_pairs(clauses, vectors, floor=0.30):
pairs = []
for i in range(len(clauses)):
for j in range(i + 1, len(clauses)):
if clauses[i]["company"] == clauses[j]["company"]:
continue
sim = cosine(vectors[i], vectors[j])
if sim >= floor:
pairs.append((sim, i, j))
return sorted(pairs, reverse=True)
The 0.30 floor throws out almost every cross-company pair, and the top of the sorted list is worth a look on its own before any clustering. The clustering step below uses a stricter cut.
Union-find turns pairs into groups. Each pair at or above 0.35 merges the two quotes' groups, and find halves the path on every lookup so the trees stay shallow. Only groups that span two or more companies are kept.
from collections import defaultdict
def clusters(clauses, pairs, threshold=0.35):
parent = list(range(len(clauses)))
def find(x):
while parent[x] != x:
parent[x] = parent[parent[x]]
x = parent[x]
return x
for sim, i, j in pairs:
if sim >= threshold:
parent[find(i)] = find(j)
groups = defaultdict(list)
for i in range(len(clauses)):
groups[find(i)].append(i)
shared = [g for g in groups.values() if len({clauses[i]["company"] for i in g}) > 1]
def size(g):
return len({clauses[i]["company"] for i in g}), len(g)
return sorted(shared, key=size, reverse=True)
That's single-linkage clustering, and it chains. When quote A matches B and B matches C, all three end up in one group even if A and C share little. In our run a broad Google clause about user conduct landed in the AI-training group through a chain of other quotes and brought several Google domains in with it. Read the members of a cluster before you give it a name.
The report prints each group's size, its clause types and the first few quotes, one per company.
def show(clauses, groups, examples=3):
for idxs in groups:
companies = {clauses[i]["company"] for i in idxs}
types = sorted({clauses[i]["type"] for i in idxs})
print(f"{len(companies)} companies, {len(idxs)} quotes, {', '.join(types)}")
seen = set()
for i in idxs:
c = clauses[i]
if c["company"] not in seen:
seen.add(c["company"])
print(f' {c["company"]:<16} "{c["quote"][:100]}"')
if len(seen) == examples:
break
clauses = load_clauses("evidence.csv")
vectors = tfidf_vectors([c["quote"] for c in clauses])
show(clauses, clusters(clauses, similar_pairs(clauses, vectors)))
On the study's data the largest group is the "robot, spider, scraper" template on 17 sites, with Netflix, Perplexity and Expedia among its first examples. Next comes the "data mining, robots or similar data gathering" construction on 13 sites, and anti-circumvention wording appears on 12.
Watch what the company column holds. Our consumer-site tier came from a domain ranking, so one owner can appear under two names, netflix.com next to netflix.net, or OpenAI next to chatgpt.com. The same-company check won't catch their identical clauses, so they count as a cross-company pair. If you want to count companies instead of sites, map domains to owners before pairing.
The anti-scraping side of the sample groups into a few shared templates, and a ban that reads like one company's own policy often has a near-twin in another company's Terms.
The full study has the scoring, the top-15 table and the robots.txt matrix of AI platforms blocking each other's crawlers. Read the AI Hypocrisy Index on the HasData blog.