# Merging overlapping scrape results without double counting: 1,869 rows in, 1,671 out > Source: > Published: 2026-11-06 02:47:01+00:00 Most of my scraping jobs don't produce one clean dataset. They produce several that overlap: the same search run with four keywords, the same category page scraped every night, two scrapers pointed at sites that list the same products. Stack them up and count, and you count some things twice. The fix is boring, which is probably why it gets skipped: merge everything, pick a key, keep one row per key. This post walks through that on a real example with real numbers, first in plain Python so you can see the logic, then with an Apify Actor I wrote for the case where the data already lives in Apify datasets. Disclosure: I built the Actor used below ([Dataset Deduplication & Merge](https://apify.com/ahmed.fouaad82/dataset-dedup-merge)) and I get paid when other people run it. This article was written with AI assistance. The numbers come from one real run on 2026-11-05 (run id below), and I ran the code against the live APIs before posting. I wanted every Hacker News story from the last 120 days about databases. The Algolia HN API is public and needs no key, but it's keyword search, so I ran four queries: `sqlite`, `postgres`, `duckdb` and `database`. A story titled "Databases in Depth: SQLite, DuckDB, PostgreSQL..." matches all four. If I just concatenate the results, it shows up four times, and any "top stories by points" chart I build later is wrong. Here's the fetch. Each query's hits go into their own Apify dataset, which is roughly what you'd have after four scraper runs: ``` python import time, requests TOKEN = "YOUR_APIFY_TOKEN" API = "https://api.apify.com/v2" H = {"Authorization": f"Bearer {TOKEN}"} now = int(time.time()) since = now - 120 * 86400 # last 120 days fields = ["objectID", "title", "url", "author", "points", "num_comments", "created_at"] dataset_ids = [] for query in ["sqlite", "postgres", "duckdb", "database"]: hits = requests.get("https://hn.algolia.com/api/v1/search", params={ "query": query, "tags": "story", "hitsPerPage": 1000, "numericFilters": f"created_at_i>{since},created_at_i<{now}", }, timeout=60).json()["hits"] rows = [{**{f: h.get(f) for f in fields}, "query": query} for h in hits] ds = requests.post(f"{API}/datasets", headers=H, timeout=60).json()["data"] requests.post(f"{API}/datasets/{ds['id']}/items", headers=H, json=rows, timeout=60).raise_for_status() dataset_ids.append(ds["id"]) print(query, len(rows), ds["id"]) ``` On 2026-11-05 that gave me 292 rows for sqlite, 646 for postgres, 165 for duckdb and 766 for database: 1,869 rows in total. The search index changes daily, so your counts will be a little different. I added a `query` field to every row on purpose. After deduplication it tells you which search found the story first, which turns out to be handy. This is the decision that matters, and it depends on what "the same thing" means for your data. `objectID` is HN's own story id. Two rows with the same id are the same story, full stop. That's the safe key.`url` is tempting, because the same blog post often gets submitted more than once under different ids. But Ask HN and Show HN text posts have no URL at all, and you don't want every URL-less story collapsed into one row. I started with `objectID`. I'll come back to `url` at the end. If all the rows fit in one list in memory (call it `all_rows`), this would be the whole job: ``` seen, unique = set(), [] for row in all_rows: if row["objectID"] not in seen: seen.add(row["objectID"]) unique.append(row) ``` It stops being that simple when the rows are spread across several datasets, there are a few million of them, you want to know how many times each key appeared, or the key is two fields together (shop + SKU, say). That's the part the Actor handles. It reads the datasets in order, keeps the key index on disk in SQLite instead of in memory, and writes one row per key with a count. Input: ``` { "datasetIds": ["", "", "", ""], "keyFields": ["objectID"], "keep": "first", "trim": true, "ignoreEmpty": true, "demoMode": false } ``` `keep: "first"` keeps the earliest occurrence in the order you list the datasets; `"last"` keeps the latest, which is what you want when later datasets are fresher snapshots. `trim` strips whitespace from key values before comparing. `ignoreEmpty` keeps rows with a missing or empty key as separate rows instead of merging them all into one. `keyFields` takes several fields if you need a composite key, and dotted paths like `product.sku` for nested values. You can paste that into the Actor page and press Start. From code, continuing the script above: ``` run_input = { "datasetIds": dataset_ids, "keyFields": ["objectID"], "keep": "first", "trim": True, "ignoreEmpty": True, "demoMode": False, } run = requests.post( f"{API}/acts/ahmed.fouaad82~dataset-dedup-merge/runs", headers=H, json=run_input, params={"waitForFinish": 60, "maxTotalChargeUsd": 0.10}, timeout=90, ).json()["data"] while run["status"] in ("READY", "RUNNING"): time.sleep(5) run = requests.get(f"{API}/actor-runs/{run['id']}", headers=H, timeout=60).json()["data"] print(run["status"], run["id"]) summary = requests.get( f"{API}/key-value-stores/{run['defaultKeyValueStoreId']}/records/SUMMARY", headers=H, timeout=60 ).json() print(summary) ``` `waitForFinish` tops out at 60 seconds on the Apify API, hence the polling loop. `maxTotalChargeUsd` is a hard spending cap for the run. My run (id `GGii3VrqS3GQZG1FW`) took 5.4 seconds and printed this summary: ``` { "processedItems": 1869, "outputItems": 1671, "uniqueItems": 1671, "duplicateItems": 198, "sourceDatasets": 4, "stopReason": "completed", "complete": true } ``` So 198 of the 1,869 rows, about 11%, were the same story found by more than one query. Not dramatic, but enough to skew any count or ranking. Each output row wraps the original record and adds three fields: ``` { "item": { "objectID": "49010949", "title": "Show HN: SyncLite – An embedded synchronization layer for SQLite and DuckDB", "url": "https://github.com/syncliteio/SyncLite", "author": "mahcha", "points": 2, "num_comments": 4, "created_at": "2026-07-22T18:06:47Z", "query": "sqlite" }, "occurrences": 4, "sourceDatasetId": "oUicy9wRLdWsHim8l", "sourceIndex": 103 } ``` `occurrences` is the number of source rows that shared the key. `sourceDatasetId` and `sourceIndex` say exactly where the kept row came from, so you can always trace a row back to its run. `occurrences` doubles as a free signal. Here it's "how many of my queries matched this story", a rough relevance score: ``` items = requests.get( f"{API}/datasets/{run['defaultDatasetId']}/items", headers=H, params={"limit": 5000}, timeout=60, ).json() repeats = sorted((i for i in items if i["occurrences"] > 1), key=lambda i: -i["occurrences"]) print(len(items), "rows,", len(repeats), "stories matched more than one query") for i in repeats[:3]: print(i["occurrences"], i["item"]["title"]) 1671 rows, 174 stories matched more than one query 4 Show HN: SyncLite – An embedded synchronization layer for SQLite and DuckDB 4 Databases in Depth – SQLite · DuckDB · PostgreSQL · MariaDB · ClickHouse · Mongo 4 Show HN: Spice 2.0 – Real-Time Analytical Query on Operational Data, Without ETL ``` The full spread was 1,497 stories seen once, 154 seen twice, 16 three times and 4 by all four queries. If you only want to audit what got merged, set `"onlyDuplicates": true` and you get just the repeated keys (174 rows here) instead of the full cleaned set. To get a flat CSV instead of nested JSON, export the output dataset with `unwind=item`, or flatten `item` yourself. Deduplicating by id removed the query overlap, but HN has another kind of duplicate: one article submitted several times by different people, each with its own id. A quick local count on the 1,671 rows: ``` python from collections import Counter urls = Counter((i["item"].get("url") or "").strip().lower() for i in items) urls.pop("", None) print(sum(urls.values()) - len(urls), "rows share a URL with an earlier row") ``` That prints 222. One ClickHouse blog post about Postgres appeared 11 times. Whether those count as duplicates depends on the question. For "which articles did people share", yes. For "which submissions got traction", no, because each submission has its own points and comments. If you want them merged, you can feed the first run's output dataset straight back in. The key is now nested under `item`, so it's a dotted path: ``` { "datasetIds": [""], "keyFields": ["item.url"], "caseInsensitive": true, "trim": true, "ignoreEmpty": true, "keep": "first" } ``` I didn't run this second pass for the post, so the 222 comes from the Python count above, not from the Actor. Two things to know before you try it. With `ignoreEmpty` on, the 114 stories without a URL stay as separate rows instead of collapsing into one. And the Actor only trims and lowercases. It doesn't normalise URLs, so `http://` vs `https://`, a trailing slash or a `utm_` parameter still count as different keys. Lowercasing can also, in theory, merge two URLs whose paths differ only in case. Clean the URLs into a separate field first if that matters to you. The Actor charges $0.02 per 1,000 source rows read, and the first 50 rows of each run are free. A run like mine would cost (1,869 − 50) × $0.00002 = $0.036. It doesn't use a browser or proxies, so it finishes in a few seconds. The Pricing tab on the [Actor page](https://apify.com/ahmed.fouaad82/dataset-dedup-merge) is the authoritative number. `POST /datasets/{id}/items` call above does that). If you run it on your own data and something merges that shouldn't (or doesn't merge when it should), open an issue on the Actor page with the input you used and I'll look at it.