{"slug": "finding-near-identical-terms-of-service-clauses-with-tf-idf-character-n-grams", "title": "Finding near-identical Terms of Service clauses with TF-IDF character n-grams", "summary": "A developer built a standard-library Python pipeline that vectorizes verified anti-scraping Terms of Service clauses as character 4-grams with TF-IDF weights and links cross-company pairs at cosine similarity of 0.35 or higher. Applied to clauses from 35 AI companies and platforms and 150 top consumer sites, the analysis surfaced a \"robot, spider, scraper\" template shared by 17 sites, with the code reproducing scikit-learn's TfidfVectorizer similarities on the study's quotes.", "body_md": "**TLDR** We pulled verified anti-scraping clauses out of the Terms of Service of 35 AI companies and platforms and 150 top consumer sites, vectorized them as character 4-grams with TF-IDF weights, and linked every cross-company pair at a cosine of 0.35 or higher. The biggest group that came out is the \"robot, spider, scraper\" template, shared by 17 sites. It's standard-library Python, and on the study's quotes it gives the same similarities as scikit-learn's `TfidfVectorizer` with a character analyzer.\n\nFor the [AI Hypocrisy Index](https://hasdata.com/blog/ai-hypocrisy-index?utm_source=devto&utm_medium=syndication&utm_campaign=ai-hypocrisy-index&utm_content=intro) we scored how hard each company's Terms ban scraping against how broadly it claims user or public data. Side by side, many of the anti-scraping clauses read like one sentence with a word swapped. Netflix's Terms say \"use any robot, spider, scraper or other automated means to access the Netflix service.\" Perplexity's say \"use any robot, spider, crawlers, scraper, or other automatic device, process, software or queries.\"\n\nWe measured the overlap with code, since there are far too many pairs to compare by eye. The input is the study's evidence table, one row per clause. A clause only got into that table when its quote turned up as an exact substring of the fetched document. That check matters for similarity work, because a paraphrase from the classifier would make two companies look more alike, or less alike, than their legal text is.\n\nEach row carries a `clause_type` whose prefix names the clause. The prefixes `AS1` to `AS7` mark the seven anti-scraping types (a general ban, an absolute ban, prior written consent, a ban on AI training, liquidated damages, anti-circumvention and API-only access), and the `EX` types on the extraction side are left out. A quote under 20 characters has only a handful of n-grams and matches too easily, so it's dropped.\n\n``` python\nimport csv\n\nANTI_SCRAPING = {\"AS1\", \"AS2\", \"AS3\", \"AS4\", \"AS5\", \"AS6\", \"AS7\"}\n\ndef load_clauses(path):\n    clauses = []\n    with open(path, encoding=\"utf-8\", newline=\"\") as f:\n        for row in csv.DictReader(f):\n            kind = row[\"clause_type\"].split(\"_\")[0]\n            quote = row[\"verbatim_quote\"].strip()\n            if kind in ANTI_SCRAPING and len(quote) >= 20:\n                clauses.append({\"company\": row[\"company\"], \"type\": kind, \"quote\": quote})\n    return clauses\n```\n\nThe loader expects a CSV with `company`, `clause_type` and `verbatim_quote` columns, and it keeps only rows whose type starts with `AS1` to `AS7`, so label your own clause table that way or edit the set. The IDF weights come from whichever quotes you load, which means the 0.30 and 0.35 cuts below were set on the study's table and are worth re-checking on yours.\n\nLegal wording drifts in ways that word tokens handle badly. \"Crawler\" and \"crawlers\" are different words, and so are \"automated\" and \"automatic\", but as character 4-grams they share most of their pieces. Lowercasing and collapsing whitespace first keeps line breaks from the source documents out of the n-grams.\n\nTF-IDF does the other half of the job. N-grams that show up in most clauses, such as the pieces of \"or\" and \"any\", get a low IDF, so the similarity between two quotes comes from the phrasing they share with few others. Each vector is L2-normalized, which turns cosine similarity into a plain dot product.\n\n``` python\nimport re\nfrom collections import Counter\nfrom math import log, sqrt\n\ndef char_ngrams(text, n=4):\n    text = re.sub(r\"\\s+\", \" \", text.lower()).strip()\n    return [text[i:i + n] for i in range(len(text) - n + 1)]\n\ndef tfidf_vectors(docs, n=4):\n    grams = [char_ngrams(d, n) for d in docs]\n    df = Counter(g for doc in grams for g in set(doc))\n    idf = {g: log((len(docs) + 1) / (df[g] + 1)) + 1 for g in df}\n    vectors = []\n    for doc in grams:\n        vec = {g: tf * idf[g] for g, tf in Counter(doc).items()}\n        norm = sqrt(sum(v * v for v in vec.values())) or 1\n        vectors.append({g: v / norm for g, v in vec.items()})\n    return vectors\n\ndef cosine(a, b):\n    if len(a) > len(b):\n        a, b = b, a\n    return sum(v * b.get(g, 0) for g, v in a.items())\n```\n\nThe smoothed IDF, `log((N + 1) / (df + 1)) + 1`, is the formula scikit-learn uses by default. On our quotes the dictionary version above and `TfidfVectorizer` agreed to floating-point precision, so if you're already using scikit-learn (`pip install scikit-learn`), five lines give the same similarity matrix. They part ways on whitespace. Scikit-learn's character analyzer only collapses runs of two or more whitespace characters, so a lone line break inside a quote stays in its n-grams.\n\n``` python\nfrom sklearn.feature_extraction.text import TfidfVectorizer\n\nclauses = load_clauses(\"evidence.csv\")\nquotes = [c[\"quote\"] for c in clauses]\nX = TfidfVectorizer(analyzer=\"char\", ngram_range=(4, 4)).fit_transform(quotes)\nsimilarity = (X @ X.T).toarray()\n```\n\nCalling `toarray()` builds an N by N matrix of floats, which comes to about 200 MB at 5,000 quotes. Past a few thousand quotes it's better to keep it sparse and read only the rows you need. To run the pairing step on it, read `similarity[i, j]` where `similar_pairs` calls `cosine(vectors[i], vectors[j])`.\n\nEvery quote is compared with every other one, and pairs from the same company are skipped. A company that repeats its own ban in two sections of its Terms doesn't tell you whether it shares a template with anyone else.\n\n``` python\ndef similar_pairs(clauses, vectors, floor=0.30):\n    pairs = []\n    for i in range(len(clauses)):\n        for j in range(i + 1, len(clauses)):\n            if clauses[i][\"company\"] == clauses[j][\"company\"]:\n                continue\n            sim = cosine(vectors[i], vectors[j])\n            if sim >= floor:\n                pairs.append((sim, i, j))\n    return sorted(pairs, reverse=True)\n```\n\nThe 0.30 floor throws out almost every cross-company pair, and the top of the sorted list is worth a look on its own before any clustering. The clustering step below uses a stricter cut.\n\nUnion-find turns pairs into groups. Each pair at or above 0.35 merges the two quotes' groups, and `find` halves the path on every lookup so the trees stay shallow. Only groups that span two or more companies are kept.\n\n``` python\nfrom collections import defaultdict\n\ndef clusters(clauses, pairs, threshold=0.35):\n    parent = list(range(len(clauses)))\n\n    def find(x):\n        while parent[x] != x:\n            parent[x] = parent[parent[x]]\n            x = parent[x]\n        return x\n\n    for sim, i, j in pairs:\n        if sim >= threshold:\n            parent[find(i)] = find(j)\n\n    groups = defaultdict(list)\n    for i in range(len(clauses)):\n        groups[find(i)].append(i)\n    shared = [g for g in groups.values() if len({clauses[i][\"company\"] for i in g}) > 1]\n\n    def size(g):\n        return len({clauses[i][\"company\"] for i in g}), len(g)\n\n    return sorted(shared, key=size, reverse=True)\n```\n\nThat's single-linkage clustering, and it chains. When quote A matches B and B matches C, all three end up in one group even if A and C share little. In our run a broad Google clause about user conduct landed in the AI-training group through a chain of other quotes and brought several Google domains in with it. Read the members of a cluster before you give it a name.\n\nThe report prints each group's size, its clause types and the first few quotes, one per company.\n\n``` python\ndef show(clauses, groups, examples=3):\n    for idxs in groups:\n        companies = {clauses[i][\"company\"] for i in idxs}\n        types = sorted({clauses[i][\"type\"] for i in idxs})\n        print(f\"{len(companies)} companies, {len(idxs)} quotes, {', '.join(types)}\")\n        seen = set()\n        for i in idxs:\n            c = clauses[i]\n            if c[\"company\"] not in seen:\n                seen.add(c[\"company\"])\n                print(f'  {c[\"company\"]:<16} \"{c[\"quote\"][:100]}\"')\n            if len(seen) == examples:\n                break\n\nclauses = load_clauses(\"evidence.csv\")\nvectors = tfidf_vectors([c[\"quote\"] for c in clauses])\nshow(clauses, clusters(clauses, similar_pairs(clauses, vectors)))\n```\n\nOn the study's data the largest group is the \"robot, spider, scraper\" template on 17 sites, with Netflix, Perplexity and Expedia among its first examples. Next comes the \"data mining, robots or similar data gathering\" construction on 13 sites, and anti-circumvention wording appears on 12.\n\nWatch what the `company` column holds. Our consumer-site tier came from a domain ranking, so one owner can appear under two names, netflix.com next to netflix.net, or OpenAI next to chatgpt.com. The same-company check won't catch their identical clauses, so they count as a cross-company pair. If you want to count companies instead of sites, map domains to owners before pairing.\n\nThe anti-scraping side of the sample groups into a few shared templates, and a ban that reads like one company's own policy often has a near-twin in another company's Terms.\n\nThe full study has the scoring, the top-15 table and the robots.txt matrix of AI platforms blocking each other's crawlers. [Read the AI Hypocrisy Index on the HasData blog](https://hasdata.com/blog/ai-hypocrisy-index?utm_source=devto&utm_medium=syndication&utm_campaign=ai-hypocrisy-index&utm_content=cta).", "url": "https://wpnews.pro/news/finding-near-identical-terms-of-service-clauses-with-tf-idf-character-n-grams", "canonical_source": "https://dev.to/hasdata_com/finding-near-identical-terms-of-service-clauses-with-tf-idf-character-n-grams-3chf", "published_at": "2026-10-22 15:04:00+00:00", "updated_at": "2026-10-22 15:13:58.712446+00:00", "lang": "en", "topics": ["ai-crawlers", "natural-language-processing", "ai-policy", "developer-tools"], "entities": ["Netflix", "Perplexity", "scikit-learn", "HasData"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/finding-near-identical-terms-of-service-clauses-with-tf-idf-character-n-grams", "markdown": "https://wpnews.pro/news/finding-near-identical-terms-of-service-clauses-with-tf-idf-character-n-grams.md", "text": "https://wpnews.pro/news/finding-near-identical-terms-of-service-clauses-with-tf-idf-character-n-grams.txt", "jsonld": "https://wpnews.pro/news/finding-near-identical-terms-of-service-clauses-with-tf-idf-character-n-grams.jsonld"}}