# 24 of the 30 duplicate groups were not duplicates

> Source: <https://dev.to/untactit/24-of-the-30-duplicate-groups-were-not-duplicates-2m56>
> Published: 2026-08-18 23:02:07+00:00

We grouped an asset database by title and found 30 groups of "duplicates." Content hashing dissolved 24 of them. Five were real. One was unjudgeable — because the rows had no content at all.

This is a field note on deduplication judgment, from auditing a store of AI-agent assets (conversations, skills, memories, instruction files) collected across many platforms. The numbers are from a live audit on 2026-08-19.

The 30 groups came from exact title matches. That is a fine way to *find candidates*. It is a terrible way to *decide*. For each group we judged on four axes instead:

Here is how the 30 groups actually broke down.

Identical content, twice in the store. Every one had a mundane cause: the same asset written under a underscore slug and a hyphen slug; the same memory saved once under an English key and once under a localized key; the same body captured under two different provenance IDs.

Real duplicates are boring. That is the point — they are the minority, and they are the only rows you may safely collapse.

Two conversations titled identically, created 45 days apart, 8% body similarity. A 213KB thread and a 27KB thread sharing a name. One pair sat at 85% similarity with a **ten-month** gap — almost certainly a person pasting an old conversation to continue it, which makes it two assets, not one.

Collapse these and you destroy history. Title reuse is normal human behavior, not corruption.

Verification probes — synthetic assets written to test a pipeline — differ only by an embedded marker. Two probes from different runs look like duplicates to every fuzzy matcher. They are separate *events*, and folding them together erases the record of one run.

If your system writes test artifacts into real storage, your dedup logic needs to know what a probe looks like. Ours does now.

The strangest finding: rows whose body was **entirely empty** — header only. Same titles as real assets, no content. All of them were created inside one window of about 8.7 minutes on a single afternoon, up to five copies of one title, seconds apart.

That pattern is not a duplicate problem. It is the fossil record of a write-path incident — something retried row creation without writing bodies. The dedup lesson: **when the only evidence you have is absence, you cannot classify.** These rows go to quarantine, not to a merge queue.

Three pairs shared the same provenance reference while holding completely unrelated content — different platforms, different topics, different everything. The field that was supposed to anchor identity had collisions.

Any pipeline that had trusted provenance alone to say "already collected, skip" would have silently dropped real assets. We found the collisions only because content hashing disagreed with the reference field — two signals, arguing, is what an audit is for.

| Verdict | Groups | Safe action |
|---|---|---|
| Real duplicate (byte-identical) | 5 | Archive one copy |
| Same title, different asset | 6 | Keep both |
| Distinct test probes | 7 | Keep both, tag as probes |
| Empty-shell rows | 4 (12 rows) | Quarantine, investigate the writer |
| Provenance collisions | 3 pairs | Re-verify at the source |

80% of what looked like duplication was not. The uncomfortable inverse also holds: a store that "has no duplicates" by title may be full of byte-identical content under different names. Both errors come from the same root — judging identity by the cheapest available key.

*Notes from building untactit — pre-launch — where asset identity across platforms is the whole job. Content hash plus provenance plus read-back verification; never title alone.*