{"slug": "diff-two-snapshots-of-a-messy-csv-in-python-no-break-spaces-duplicate-rows-and", "title": "Diff Two Snapshots of a Messy CSV in Python: No-Break Spaces, Duplicate Rows and Renamed Keys", "summary": "A developer maintaining the VPS Pricing Index published a Python-based approach for diffing successive git snapshots of a hand-maintained CSV, handling thousands separators, currency symbols, footnote daggers, em dashes and no-break spaces that otherwise produce false added/removed rows. The method uses three helpers: a regex-based money parser that returns None for unpublished values, NFKC text normalization to collapse invisible character differences, and a loader that locates the real header below a human-readable title row.", "body_md": "If you build anything on top of a public dataset, sooner or later a new version lands and you have to answer a simple question: what changed? With a hand-maintained CSV, that question is harder than it looks. Keys get renamed, cells carry footnote symbols, and one invisible character can turn an unchanged row into an \"added\" and a \"removed\" at the same time.\n\nThis tutorial builds a small Python diff for two git snapshots of the [VPS Pricing Index](https://vpshostingradar.com/vps-pricing-index/), a hosting-price dataset I maintain. I drafted the article with AI assistance; every command was run against the real repository, and the outputs below are copied from those runs and checked against the files by hand. One disclosure: VPS Hosting Radar earns affiliate commissions on some of its pages. This post has no affiliate links.\n\nThe dataset lives in a [public GitHub repository](https://github.com/vpshostingradar/vps-pricing-data) under CC BY 4.0. `pricing.csv` has three commits, and git keeps every version:\n\n```\ngit clone https://github.com/vpshostingradar/vps-pricing-data.git\ncd vps-pricing-data\ngit log --oneline -- pricing.csv\n# 39a4cad Update pricing.csv\n# 53b7d49 Initial dataset: 16 plans across 13 providers, collected 2026-08-11\n# 8e6fc61 Create pricing.csv\n\ngit show 53b7d49:pricing.csv > aug.csv   # verified 2026-08-11\ngit show 39a4cad:pricing.csv > sep.csv   # verified 2026-09-15\n```\n\n`git show <commit>:<path>` writes a file exactly as it was at that commit, so you don't need to check anything out. (The oldest commit, `8e6fc61`, is a header-only placeholder with a different snake_case schema and no data rows, so I left it out.)\n\nOpen `aug.csv` and you'll see the things that break naive tooling:\n\n`\"2,520.00\"` with a thousands separator, `€4.40‡` with a currency sign and a footnote dagger, `€ 5.50` with a space, and `—` (an em dash) meaning \"not published.\"`Cloud VPS 4` and `(EUR)`. It looks identical on screen.\nNone of this is unusual. It's what happens when people copy values out of pricing pages and spreadsheets by hand. You want a diff that survives it.\n\nStart with three helpers: one that pulls a number out of whatever is in a money cell, one that normalizes text so invisible differences stop mattering, and a loader that finds the real header.\n\n``` python\nimport csv\nimport re\nimport sys\nimport unicodedata\nfrom collections import Counter\n\nMONEY = re.compile(r\"-?\\d+(?:\\.\\d+)?\")\n\ndef money(cell):\n    \"\"\"'2,520.00' -> 2520.0, '€ 5.50' -> 5.5, '35.00‡' -> 35.0, '—' -> None.\"\"\"\n    cell = (cell or \"\").replace(\",\", \"\").strip()\n    m = MONEY.search(cell)\n    return float(m.group()) if m else None\n\ndef norm(text):\n    \"\"\"NFKC turns U+00A0 (no-break space) into a plain space; then collapse runs.\"\"\"\n    return \" \".join(unicodedata.normalize(\"NFKC\", text).split())\n\ndef load(path):\n    with open(path, newline=\"\", encoding=\"utf-8\") as f:\n        lines = list(csv.reader(f))\n    # Row 0 is a human-readable title, not the header. Find the real header.\n    start = next(i for i, row in enumerate(lines) if row and row[0] == \"Provider\")\n    header = lines[start]\n    return [dict(zip(header, r)) for r in lines[start + 1:] if any(r)]\n```\n\nA few choices here are deliberate.\n\n`money()` returns `None` for an em dash rather than `0`. \"The vendor didn't publish a monthly price\" and \"the monthly price is zero\" are different facts, and the diff should report a move from `None` to `26.99` as new information, not as a $26.99 price rise.\n\nThe regex ignores the currency symbol, which is safe only because currency doesn't change within a row between snapshots. If you're diffing a dataset where a row could switch from USD to EUR, keep the symbol as a separate field and compare it too.\n\n`norm()` uses NFKC normalization. Among other things, NFKC maps U+00A0 to an ordinary space. `str.split()` with no arguments already treats U+00A0 as whitespace in Python 3, so the `split`/` join` pass alone would catch this particular character. NFKC also handles full-width digits and similar compatibility characters, which turn up when values get pasted from some PDFs and from CJK input methods.\n\nA diff needs a key. Here the natural key is `(Provider, Plan)`. Two things can go wrong with it: duplicates within one file, and keys that change between files because the maintainer swapped one plan for another.\n\n``` python\ndef keyed(rows, label):\n    keys = [(norm(r[\"Provider\"]), norm(r[\"Plan\"])) for r in rows]\n    for key, n in Counter(keys).items():\n        if n > 1:\n            print(f\"[{label}] duplicate key {key} appears {n}x\")\n    out = {}\n    for key, r in zip(keys, rows):\n        out.setdefault(key, r)  # keep first occurrence\n    return out\n\n# Row replacements are editorial decisions, so they come from the dataset's\n# own notes/changelog, not from fuzzy matching. new key -> old key.\nREPLACES = {\n    (\"Hostinger\", \"Web Premium\"): (\"Hostinger\", \"Cloud Startup\"),\n    (\"Hostinger\", \"KVM 1\"): (\"Hostinger\", \"KVM 2\"),\n    (\"InMotion\", \"VPS 4 vCPU\"): (\"InMotion\", \"VPS 4 (Most Popular)\"),\n    (\"ScalaHosting\", \"Managed Cloud #1\"): (\"ScalaHosting\", \"Managed Build #1\"),\n}\n```\n\n`keyed()` warns about duplicates instead of silently overwriting them. Keeping the first occurrence is fine here because the duplicates are identical; if they weren't, you'd want the script to stop.\n\nThe `REPLACES` map is the part people are tempted to automate. Don't. Hostinger has two rows in each snapshot, so \"match by provider\" is ambiguous. Fuzzy string matching would happily pair `KVM 2` with `KVM 1`, which happens to be right, but for the wrong reason: that pairing is true because the September notes column says \"Replaces KVM 2 (8 GB), which overshot the 4 GB baseline,\" not because the strings are similar. Every entry in the map above comes from a note in the September file or from the repository's `CHANGELOG.md`. When the next snapshot arrives, you read the changelog and update the map. It takes two minutes and you'll trust the output afterwards.\n\n```\nFIELDS = {\n    \"First term $/mo\": \"intro\",\n    \"Prepay months\": \"prepay\",\n    \"Monthly $/mo\": \"monthly\",\n    \"Renewal $/mo\": \"renewal\",\n    \"36-month total $\": \"total_36\",\n}\n\ndef compare(old, new):\n    matched = set()\n    for key, n in new.items():\n        okey = key if key in old else REPLACES.get(key)\n        if okey not in old:\n            print(f\"+ added    {key[0]} {key[1]}\")\n            continue\n        matched.add(okey)\n        tag = \"same plan\" if okey == key else f\"replaces {okey[1]}\"\n        changes = []\n        for col, short in FIELDS.items():\n            a, b = money(old[okey].get(col)), money(n.get(col))\n            if a != b:\n                changes.append(f\"{short} {a} -> {b}\")\n        if changes:\n            print(f\"~ changed  {key[0]} {key[1]} ({tag}): \" + \"; \".join(changes))\n    for okey in old.keys() - matched:\n        print(f\"- removed  {okey[0]} {okey[1]}\")\n\nif __name__ == \"__main__\":\n    old = keyed(load(sys.argv[1]), \"old\")\n    new = keyed(load(sys.argv[2]), \"new\")\n    print(f\"old: {len(old)} unique plans, new: {len(new)} unique plans\\n\")\n    compare(old, new)\n```\n\nSave the three blocks as `snapdiff.py` and run `python3 snapdiff.py aug.csv sep.csv`. Here's the output, with one `changed` line removed for length (marked `…`):\n\n```\n[old] duplicate key ('Liquid Web', 'Managed VPS 4 GB') appears 2x\n[old] duplicate key ('Contabo', 'Cloud VPS 4 (EUR)') appears 2x\nold: 16 unique plans, new: 19 unique plans\n\n~ changed  SiteGround StartUp (same plan): monthly 24.99 -> 17.99\n~ changed  Hostinger Web Premium (replaces Cloud Startup): intro 7.99 -> 2.99; prepay 48.0 -> 12.0; monthly 27.99 -> 11.99; renewal 25.99 -> 11.99; total_36 287.64 -> 323.64\n~ changed  Hostinger KVM 1 (replaces KVM 2): intro 8.79 -> 6.49; monthly 24.49 -> 19.49; renewal 14.99 -> 19.49; total_36 390.84 -> 389.64\n~ changed  InMotion VPS 4 vCPU (replaces VPS 4 (Most Popular)): intro 10.99 -> 13.99; monthly None -> 26.99; total_36 587.64 -> 659.64\n~ changed  ScalaHosting Managed Cloud #1 (replaces Managed Build #1): prepay 36.0 -> 1.0; total_36 1078.2 -> 1953.2\n+ added    ScalaHosting Unmanaged Config #1\n~ changed  WP Engine Startup (same plan): intro 30.0 -> 28.0; total_36 1200.0 -> 1176.0\n~ changed  Contabo Cloud VPS 4 (EUR) (same plan): intro 4.4 -> 5.28; monthly None -> 6.6; renewal 5.5 -> 6.6; total_36 None -> 205.92\n…\n+ added    SiteGround Cloud Jump Start (4 CPU / 8 GB / 40 GB)\n+ added    Cloudways Autonomous Growth\n```\n\nThe omitted line is the Rocket Starter row, where `prepay` goes from 1 to 12.\n\nSixteen unique plans become nineteen, with three real additions and nothing removed. That matches the changelog, which lists exactly those three as new rows.\n\nTo see what normalization buys, replace the body of `norm()` with `return text.strip()` and run it again. The `added`/` removed` lines become:\n\n```\n+ added    ScalaHosting Unmanaged Config #1\n+ added    Contabo Cloud VPS 4 (EUR)\n+ added    Cloudways DigitalOcean 4 GB Basic (premium CPU)\n+ added    SiteGround Cloud Jump Start (4 CPU / 8 GB / 40 GB)\n+ added    Cloudways Autonomous Growth\n- removed  Contabo Cloud VPS 4 (EUR)\n- removed  Cloudways DigitalOcean 4 GB Basic (premium CPU)\n```\n\nThat's seven lines where the true answer is three. Two unchanged plans show up as churn because their August names contain a no-break space. The printed names look identical in the terminal, which is exactly what makes this bug hard to spot. Drop the `REPLACES` map as well and four more plans would appear as removed-plus-added pairs, leaving you with a report that's mostly noise.\n\nHere's the part I think matters most. The script reports that values changed, but it can't tell a vendor price change from a correction in the dataset, and in this pair of snapshots most of the movement is corrections.\n\n`monthly 24.99 -> 17.99`` intro 10.99 -> 13.99`\nThe rest (WP Engine's intro rate, ScalaHosting's prepay term, the Rocket prepay term) aren't explained line by line in the changelog. For those, the diff gives you a short list of rows to check against each vendor's page, and the vendor page is the tiebreaker.\n\nThat's a good reason to keep editorial context, meaning notes and a changelog, next to any hand-collected dataset. A diff turns two hundred cells into ten lines worth reading. A human still has to say what each line means. I wrote about the [collection process and the mistakes that shaped it](https://medium.com/@vpshostingradar/how-i-hand-collected-pricing-for-16-hosting-plans-and-the-mistake-that-almost-broke-the-dataset-7e67e008e06b) on Medium, if you want the background.\n\nThe script is short enough to adapt rather than configure:\n\n`keyed()` to whatever uniquely identifies a row in your data. If there's an ID column, use it. The current site export at `/data/vps-pricing-index.csv` has a stable `id` column (`hostinger-kvm-1`, `siteground-startup` and so on), which removes the whitespace problem entirely. You'd still map deliberately replaced plans by hand.`REPLACES` small and sourced.`print` in The site's [tools page](https://vpshostingradar.com/tools/) lists the CSV and JSON exports if you want the current snapshot instead of the git history. Prices in both are dated (the latest verification was 2026-09-15), so check the vendor's page before you act on any of them.\n\n*Carter Wang is a serial founder who has bought hosting and run production stacks for more than a decade. He maintains the VPS Pricing Index at [VPS Hosting Radar](https://vpshostingradar.com/).*", "url": "https://wpnews.pro/news/diff-two-snapshots-of-a-messy-csv-in-python-no-break-spaces-duplicate-rows-and", "canonical_source": "https://dev.to/carterwang/diff-two-snapshots-of-a-messy-csv-in-python-no-break-spaces-duplicate-rows-and-renamed-keys-2kf7", "published_at": "2026-10-30 05:18:02+00:00", "updated_at": "2026-10-30 05:19:14.595601+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["VPS Pricing Index", "VPS Hosting Radar", "GitHub", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/diff-two-snapshots-of-a-messy-csv-in-python-no-break-spaces-duplicate-rows-and", "markdown": "https://wpnews.pro/news/diff-two-snapshots-of-a-messy-csv-in-python-no-break-spaces-duplicate-rows-and.md", "text": "https://wpnews.pro/news/diff-two-snapshots-of-a-messy-csv-in-python-no-break-spaces-duplicate-rows-and.txt", "jsonld": "https://wpnews.pro/news/diff-two-snapshots-of-a-messy-csv-in-python-no-break-spaces-duplicate-rows-and.jsonld"}}