{"slug": "how-i-check-duplicate-ids-and-missing-cells-in-csv-with-python", "title": "How I check duplicate IDs and missing cells in CSV with Python", "summary": "A developer created a Python script using only the standard library to check CSV files for duplicate IDs and missing cells. The script returns observable facts like row count, column names, duplicate ID count, and missing cell counts per column. The approach intentionally separates missing-value checks from numeric validation to avoid overclaiming.", "body_md": "AI disclosure:This article was prepared with AI assistance and reviewed against the runnable sample and its reproducibility test.\n\nCSV quality problems often appear before a full validation framework is necessary. A quick first pass can answer three useful questions:\n\nThe following example uses only Python's standard library and synthetic data.\n\n```\nid,amount,qty,category\nA001,10.00,1,books\nA002,5.50,2,toys\nA003,,3,books\nA001,10.00,1,books\nA004,7.25,4,home\nA005,abc,2,toys\nA006,0,1,accessories\n```\n\nThere are seven rows, one repeated ID, and one blank value in `amount`\n\n. Notice that `abc`\n\nis not blank—it is a separate numeric-validation problem. Keeping those concepts separate prevents a simple missing-value check from making claims it cannot support.\n\n``` python\nimport csv\nfrom collections import Counter\nfrom pathlib import Path\n\ndef profile_csv(path: Path, id_field: str = \"id\") -> dict:\n    with path.open(\"r\", encoding=\"utf-8-sig\", newline=\"\") as handle:\n        reader = csv.DictReader(handle)\n        rows = [dict(row) for row in reader]\n        columns = list(reader.fieldnames or [])\n\n    id_counts = Counter(\n        row.get(id_field, \"\").strip()\n        for row in rows\n        if row.get(id_field, \"\").strip()\n    )\n    duplicate_id_count = sum(\n        count - 1 for count in id_counts.values() if count > 1\n    )\n    missing_cells = {\n        column: sum(1 for row in rows if not str(row.get(column, \"\")).strip())\n        for column in columns\n    }\n\n    return {\n        \"row_count\": len(rows),\n        \"columns\": columns,\n        \"duplicate_id_count\": duplicate_id_count,\n        \"missing_cells\": missing_cells,\n    }\n```\n\nFor the sample above, the result is:\n\n```\n{\n  \"row_count\": 7,\n  \"columns\": [\"id\", \"amount\", \"qty\", \"category\"],\n  \"duplicate_id_count\": 1,\n  \"missing_cells\": {\n    \"id\": 0,\n    \"amount\": 1,\n    \"qty\": 0,\n    \"category\": 0\n  }\n}\n```\n\nA small script should state its limits clearly. This example does **not**:\n\n`abc`\n\nas an invalid number;Those tasks require explicit rules and deeper tests. The useful design lesson is to return observable facts first, then add stricter behavior only when the data contract is known.\n\nThe runnable preview repository contains the synthetic CSV, expected JSON response, and a no-dependency test: [https://github.com/Kalamari0227/eidolon-csv-quality-preview](https://github.com/Kalamari0227/eidolon-csv-quality-preview)\n\nThe expanded local HTTP API package, with numeric summaries and default/strict modes, is available here: [https://ethanlee20.gumroad.com/l/dmuomd](https://ethanlee20.gumroad.com/l/dmuomd)", "url": "https://wpnews.pro/news/how-i-check-duplicate-ids-and-missing-cells-in-csv-with-python", "canonical_source": "https://dev.to/kalamari0227/how-i-check-duplicate-ids-and-missing-cells-in-csv-with-python-4fha", "published_at": "2026-07-17 22:35:05+00:00", "updated_at": "2026-07-17 23:30:04.687771+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Python", "GitHub", "Kalamari0227", "Ethan Lee"], "alternates": {"html": "https://wpnews.pro/news/how-i-check-duplicate-ids-and-missing-cells-in-csv-with-python", "markdown": "https://wpnews.pro/news/how-i-check-duplicate-ids-and-missing-cells-in-csv-with-python.md", "text": "https://wpnews.pro/news/how-i-check-duplicate-ids-and-missing-cells-in-csv-with-python.txt", "jsonld": "https://wpnews.pro/news/how-i-check-duplicate-ids-and-missing-cells-in-csv-with-python.jsonld"}}