Finding near-identical Terms of Service clauses with TF-IDF character n-grams

A developer built a standard-library Python pipeline that vectorizes verified anti-scraping Terms of Service clauses as character 4-grams with TF-IDF weights and links cross-company pairs at cosine similarity of 0.35 or higher. Applied to clauses from 35 AI companies and platforms and 150 top consumer sites, the analysis surfaced a "robot, spider, scraper" template shared by 17 sites, with the code reproducing scikit-learn's TfidfVectorizer similarities on the study's quotes.

TLDR We pulled verified anti-scraping clauses out of the Terms of Service of 35 AI companies and platforms and 150 top consumer sites, vectorized them as character 4-grams with TF-IDF weights, and linked every cross-company pair at a cosine of 0.35 or higher. The biggest group that came out is the "robot, spider, scraper" template, shared by 17 sites. It's standard-library Python, and on the study's quotes it gives the same similarities as scikit-learn's TfidfVectorizer with a character analyzer. For the AI Hypocrisy Index https://hasdata.com/blog/ai-hypocrisy-index?utm source=devto&utm medium=syndication&utm campaign=ai-hypocrisy-index&utm content=intro we scored how hard each company's Terms ban scraping against how broadly it claims user or public data. Side by side, many of the anti-scraping clauses read like one sentence with a word swapped. Netflix's Terms say "use any robot, spider, scraper or other automated means to access the Netflix service." Perplexity's say "use any robot, spider, crawlers, scraper, or other automatic device, process, software or queries." We measured the overlap with code, since there are far too many pairs to compare by eye. The input is the study's evidence table, one row per clause. A clause only got into that table when its quote turned up as an exact substring of the fetched document. That check matters for similarity work, because a paraphrase from the classifier would make two companies look more alike, or less alike, than their legal text is. Each row carries a clause type whose prefix names the clause. The prefixes AS1 to AS7 mark the seven anti-scraping types a general ban, an absolute ban, prior written consent, a ban on AI training, liquidated damages, anti-circumvention and API-only access , and the EX types on the extraction side are left out. A quote under 20 characters has only a handful of n-grams and matches too easily, so it's dropped. python import csv ANTI SCRAPING = {"AS1", "AS2", "AS3", "AS4", "AS5", "AS6", "AS7"} def load clauses path : clauses = with open path, encoding="utf-8", newline="" as f: for row in csv.DictReader f : kind = row "clause type" .split " " 0 quote = row "verbatim quote" .strip if kind in ANTI SCRAPING and len quote = 20: clauses.append {"company": row "company" , "type": kind, "quote": quote} return clauses The loader expects a CSV with company , clause type and verbatim quote columns, and it keeps only rows whose type starts with AS1 to AS7 , so label your own clause table that way or edit the set. The IDF weights come from whichever quotes you load, which means the 0.30 and 0.35 cuts below were set on the study's table and are worth re-checking on yours. Legal wording drifts in ways that word tokens handle badly. "Crawler" and "crawlers" are different words, and so are "automated" and "automatic", but as character 4-grams they share most of their pieces. Lowercasing and collapsing whitespace first keeps line breaks from the source documents out of the n-grams. TF-IDF does the other half of the job. N-grams that show up in most clauses, such as the pieces of "or" and "any", get a low IDF, so the similarity between two quotes comes from the phrasing they share with few others. Each vector is L2-normalized, which turns cosine similarity into a plain dot product. python import re from collections import Counter from math import log, sqrt def char ngrams text, n=4 : text = re.sub r"\s+", " ", text.lower .strip return text i:i + n for i in range len text - n + 1 def tfidf vectors docs, n=4 : grams = char ngrams d, n for d in docs df = Counter g for doc in grams for g in set doc idf = {g: log len docs + 1 / df g + 1 + 1 for g in df} vectors = for doc in grams: vec = {g: tf idf g for g, tf in Counter doc .items } norm = sqrt sum v v for v in vec.values or 1 vectors.append {g: v / norm for g, v in vec.items } return vectors def cosine a, b : if len a len b : a, b = b, a return sum v b.get g, 0 for g, v in a.items The smoothed IDF, log N + 1 / df + 1 + 1 , is the formula scikit-learn uses by default. On our quotes the dictionary version above and TfidfVectorizer agreed to floating-point precision, so if you're already using scikit-learn pip install scikit-learn , five lines give the same similarity matrix. They part ways on whitespace. Scikit-learn's character analyzer only collapses runs of two or more whitespace characters, so a lone line break inside a quote stays in its n-grams. python from sklearn.feature extraction.text import TfidfVectorizer clauses = load clauses "evidence.csv" quotes = c "quote" for c in clauses X = TfidfVectorizer analyzer="char", ngram range= 4, 4 .fit transform quotes similarity = X @ X.T .toarray Calling toarray builds an N by N matrix of floats, which comes to about 200 MB at 5,000 quotes. Past a few thousand quotes it's better to keep it sparse and read only the rows you need. To run the pairing step on it, read similarity i, j where similar pairs calls cosine vectors i , vectors j . Every quote is compared with every other one, and pairs from the same company are skipped. A company that repeats its own ban in two sections of its Terms doesn't tell you whether it shares a template with anyone else. python def similar pairs clauses, vectors, floor=0.30 : pairs = for i in range len clauses : for j in range i + 1, len clauses : if clauses i "company" == clauses j "company" : continue sim = cosine vectors i , vectors j if sim = floor: pairs.append sim, i, j return sorted pairs, reverse=True The 0.30 floor throws out almost every cross-company pair, and the top of the sorted list is worth a look on its own before any clustering. The clustering step below uses a stricter cut. Union-find turns pairs into groups. Each pair at or above 0.35 merges the two quotes' groups, and find halves the path on every lookup so the trees stay shallow. Only groups that span two or more companies are kept. python from collections import defaultdict def clusters clauses, pairs, threshold=0.35 : parent = list range len clauses def find x : while parent x = x: parent x = parent parent x x = parent x return x for sim, i, j in pairs: if sim = threshold: parent find i = find j groups = defaultdict list for i in range len clauses : groups find i .append i shared = g for g in groups.values if len {clauses i "company" for i in g} 1 def size g : return len {clauses i "company" for i in g} , len g return sorted shared, key=size, reverse=True That's single-linkage clustering, and it chains. When quote A matches B and B matches C, all three end up in one group even if A and C share little. In our run a broad Google clause about user conduct landed in the AI-training group through a chain of other quotes and brought several Google domains in with it. Read the members of a cluster before you give it a name. The report prints each group's size, its clause types and the first few quotes, one per company. python def show clauses, groups, examples=3 : for idxs in groups: companies = {clauses i "company" for i in idxs} types = sorted {clauses i "type" for i in idxs} print f"{len companies } companies, {len idxs } quotes, {', '.join types }" seen = set for i in idxs: c = clauses i if c "company" not in seen: seen.add c "company" print f' {c "company" :<16} "{c "quote" :100 }"' if len seen == examples: break clauses = load clauses "evidence.csv" vectors = tfidf vectors c "quote" for c in clauses show clauses, clusters clauses, similar pairs clauses, vectors On the study's data the largest group is the "robot, spider, scraper" template on 17 sites, with Netflix, Perplexity and Expedia among its first examples. Next comes the "data mining, robots or similar data gathering" construction on 13 sites, and anti-circumvention wording appears on 12. Watch what the company column holds. Our consumer-site tier came from a domain ranking, so one owner can appear under two names, netflix.com next to netflix.net, or OpenAI next to chatgpt.com. The same-company check won't catch their identical clauses, so they count as a cross-company pair. If you want to count companies instead of sites, map domains to owners before pairing. The anti-scraping side of the sample groups into a few shared templates, and a ban that reads like one company's own policy often has a near-twin in another company's Terms. The full study has the scoring, the top-15 table and the robots.txt matrix of AI platforms blocking each other's crawlers. Read the AI Hypocrisy Index on the HasData blog https://hasdata.com/blog/ai-hypocrisy-index?utm source=devto&utm medium=syndication&utm campaign=ai-hypocrisy-index&utm content=cta .