Merging overlapping scrape results without double counting: 1,869 rows in, 1,671 out A developer built an Apify Actor, Dataset Deduplication & Merge, that merges overlapping scrape results by keeping one row per key, using a SQLite-backed on-disk index instead of in-memory sets. In a real run on 2026-11-05, four Hacker News Algolia keyword queries returned 1,869 rows that deduplicated to 1,671 unique stories using HN's objectID as the key. The developer notes that URL is a tempting but unsafe key because Ask HN and Show HN text posts have no URL. Most of my scraping jobs don't produce one clean dataset. They produce several that overlap: the same search run with four keywords, the same category page scraped every night, two scrapers pointed at sites that list the same products. Stack them up and count, and you count some things twice. The fix is boring, which is probably why it gets skipped: merge everything, pick a key, keep one row per key. This post walks through that on a real example with real numbers, first in plain Python so you can see the logic, then with an Apify Actor I wrote for the case where the data already lives in Apify datasets. Disclosure: I built the Actor used below Dataset Deduplication & Merge https://apify.com/ahmed.fouaad82/dataset-dedup-merge and I get paid when other people run it. This article was written with AI assistance. The numbers come from one real run on 2026-11-05 run id below , and I ran the code against the live APIs before posting. I wanted every Hacker News story from the last 120 days about databases. The Algolia HN API is public and needs no key, but it's keyword search, so I ran four queries: sqlite , postgres , duckdb and database . A story titled "Databases in Depth: SQLite, DuckDB, PostgreSQL..." matches all four. If I just concatenate the results, it shows up four times, and any "top stories by points" chart I build later is wrong. Here's the fetch. Each query's hits go into their own Apify dataset, which is roughly what you'd have after four scraper runs: python import time, requests TOKEN = "YOUR APIFY TOKEN" API = "https://api.apify.com/v2" H = {"Authorization": f"Bearer {TOKEN}"} now = int time.time since = now - 120 86400 last 120 days fields = "objectID", "title", "url", "author", "points", "num comments", "created at" dataset ids = for query in "sqlite", "postgres", "duckdb", "database" : hits = requests.get "https://hn.algolia.com/api/v1/search", params={ "query": query, "tags": "story", "hitsPerPage": 1000, "numericFilters": f"created at i {since},created at i<{now}", }, timeout=60 .json "hits" rows = { {f: h.get f for f in fields}, "query": query} for h in hits ds = requests.post f"{API}/datasets", headers=H, timeout=60 .json "data" requests.post f"{API}/datasets/{ds 'id' }/items", headers=H, json=rows, timeout=60 .raise for status dataset ids.append ds "id" print query, len rows , ds "id" On 2026-11-05 that gave me 292 rows for sqlite, 646 for postgres, 165 for duckdb and 766 for database: 1,869 rows in total. The search index changes daily, so your counts will be a little different. I added a query field to every row on purpose. After deduplication it tells you which search found the story first, which turns out to be handy. This is the decision that matters, and it depends on what "the same thing" means for your data. objectID is HN's own story id. Two rows with the same id are the same story, full stop. That's the safe key. url is tempting, because the same blog post often gets submitted more than once under different ids. But Ask HN and Show HN text posts have no URL at all, and you don't want every URL-less story collapsed into one row. I started with objectID . I'll come back to url at the end. If all the rows fit in one list in memory call it all rows , this would be the whole job: seen, unique = set , for row in all rows: if row "objectID" not in seen: seen.add row "objectID" unique.append row It stops being that simple when the rows are spread across several datasets, there are a few million of them, you want to know how many times each key appeared, or the key is two fields together shop + SKU, say . That's the part the Actor handles. It reads the datasets in order, keeps the key index on disk in SQLite instead of in memory, and writes one row per key with a count. Input: { "datasetIds": "