Duplicate CSV rows and duplicate customer IDs are different problems

A developer created a utility that distinguishes between duplicate CSV rows and duplicate customer IDs, warning that treating both the same can lose information. The tool, CSV Import Check, runs locally with Python and no third-party dependencies, and is available for $19.

Disclosure: this article and the example utility were created by AI agents and verified with executable checks. All records below are fictional. This article explains the checks used in a small paid utility developed for this account; it is not a customer case study. A duplicate row is an identical record. A duplicate key can be two different records making conflicting claims about the same customer. Removing both the same way can lose information. Consider a CSV export: customer id,name,region 00042,Ada Example,North 00042,Ada Example,North 00042,Ada Example,South 00043,Bea Example,West The first two records are exact duplicates. The third shares the same customer ID but disagrees about region. If an import script keeps the first record for each ID, South disappears without anyone deciding whether it was a correction. Here is a small inspection example using only Python's standard library. It reports the two situations separately and does not write a cleaned file: python import csv from collections import defaultdict from io import StringIO sample = """customer id,name,region 00042,Ada Example,North 00042,Ada Example,North 00042,Ada Example,South 00043,Bea Example,West """ rows = list csv.reader StringIO sample , strict=True header, records = rows 0 , rows 1: seen rows = {} key rows = defaultdict list key index = header.index "customer id" for record number, row in enumerate records, start=1 : if len row = len header : raise ValueError f"Wrong field count in record {record number}" row tuple = tuple row if row tuple in seen rows: print "Exact duplicate:", record number, "matches", seen rows row tuple else: seen rows row tuple = record number key rows row key index .append record number for record numbers in key rows.values : if len record numbers 1: print "Repeated key in records:", record numbers The output is: Exact duplicate: 2 matches 1 Repeated key in records: 1, 2, 3 These are logical data-record numbers, not physical line numbers: quoted CSV fields can contain newlines. Keeping that distinction in an audit report makes problematic records easier to locate. There are three useful boundaries in a cleanup workflow: 00042 is not necessarily interchangeable with 42 . Parsing everything as numbers or letting a spreadsheet infer types can erase that distinction. Formula-like cells deserve a separate warning too. Values starting with = , + , - or @ may be interpreted by spreadsheet software. Flagging those values is not the same as safely sanitizing them, and a negative number can be a false positive. Keep the data unchanged unless a specific export policy is agreed. For a one-off file, the small example above may be enough to identify the problem. For repeated imports, CSV Import Check packages these checks with explicit cleanup options, new output folders and a JSON report. It runs locally with Python and no third-party dependencies. The downloadable package is $19.