{"slug": "i-paginated-by-100-and-lost-39-of-422-rows-at-99-and-101-nothing-was-missing", "title": "I paginated by 100 and lost 39 of 422 rows. At 99 and 101, nothing was missing.", "summary": "A developer investigating a DEV challenge discovered that paginating by 100 caused 39 of 422 rows to be lost, while paginating by 99 or 101 returned all rows. The bug was traced to the API's handling of page sizes, and the developer also found that a naive data collection approach silently dropped 277 of 412 bodies due to 'Retry later' responses, leading to incorrect statistics. A strict collector with retries and backoff recovered all data, and the developer used Gemini to identify that the completeness assertion only validated against the listing, not the full dataset.", "body_md": "*This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.*\n\nI wanted one number: what share of the entries in this challenge declare a particular prize category. It decides whether writing another entry is worth the evening.\n\nI got that number wrong three times in one morning. Every wrong answer looked finished.\n\n**First wrong answer: 19.** The listing endpoint returns titles and tags but not bodies, and the prize category is declared by a heading *inside* the body. So I counted titles containing \"gemini\" or \"google\". That found 19 of the 66 entries that actually carry the heading — and one of the 47 it missed was my own most recent entry, whose title mentions neither word. I had been quoting that proxy for days.\n\n**Second wrong answer: 23.9%.** So I fetched all 412 bodies, six at a time, no delay. My loader was this:\n\n```\narts = []\nfor f in dir.glob(\"*.json\"):\n    try:\n        a = json.loads(f.read_text())\n        if a.get(\"body_markdown\"):\n            arts.append(a)\n    except Exception:\n        bad += 1\n```\n\n277 of the 412 responses had the body `Retry later`\n\n. Two words, plain text, where JSON was expected. The `except`\n\nbranch counted them and moved on, and I printed a clean table from 134 items:\n\n```\n                items   coverage   category share\nnaive             134      32.5%          23.9%\ncomplete          411     100.0%          18.5%\n```\n\n5.4 points off, with nothing anomalous in it. **The number was not noisy. It was wrong, and it was plausible** — which is the only combination that actually costs you anything.\n\nThe third wrong answer is the interesting one, and I only found it because I asked something else to look at my fix. That's further down.\n\nThe loader has no bug in it. It does exactly what it says. The bug is in what it *means*: an exception during parse was being treated as evidence about the item, when it was evidence about the transport.\n\n``` python\ndef collect_strict(server, ids, max_rounds=8):\n    \"\"\"Same fetch, two extra rules: unparsed is not absent, and slow down on refusal.\"\"\"\n    done, pending, rounds = {}, list(ids), 0\n    while pending and rounds < max_rounds:\n        rounds += 1\n        still = []\n        for i in pending:\n            rec = parse(server.get(i))\n            if rec is None:\n                still.append(i)\n            else:\n                done[i] = rec\n        pending = still\n        server.cooldown //= 2          # back off: fewer requests per unit time\n    missing = [i for i in ids if i not in done]\n    if missing:\n        raise RuntimeError(f\"{len(missing)} of {len(ids)} ids never resolved\")\n    return [done[i] for i in ids], rounds\n```\n\nAgainst the real API — 0.45s between requests, doubling to a 2s ceiling — that recovered all 277 in four rounds: **134, then 77, then 63, then 3, then zero unresolved**. One id stayed unresolved and turned out to be a genuine `404`\n\n(an entry deleted between the listing call and the fetch), so the assertion permits *resolved-as-404* alongside *parsed*.\n\nThe offline reproducer at the end of this post runs the same two collectors against a fake server that refuses in bursts:\n\n```\n                       items   coverage   in category    share\nground truth             412     100.0%            97    23.5%\nnaive collector          157      38.1%            39    24.8%\nstrict collector         412     100.0%            97    23.5%\n```\n\nNote how boring the naive row is. 24.8% against a true 23.5%. Nobody looks twice at that.\n\nWith the numbers in hand, I handed the whole thing to **Gemini** and asked, among other things: *what can still silently truncate the dataset underneath my completeness assertion?*\n\nIts first-ranked answer was that my assertion validates the fetched set against **the listing**, and says nothing about whether the listing is complete.\n\nI went to check. This is the same endpoint, same tag, same minute, walked page by page until an empty page:\n\n```\n per_page  pages  unique ids   note\n       25     17         422   empty page 18\n       50                      non-JSON at page 10: 'Retry later'\n       75      6         422   empty page 7\n       99      5         422   empty page 6\n      100      4         383   empty page 5\n      101      5         422   empty page 6\n      125      4         422   empty page 5\n      150      3         422   empty page 4\n      200      3         422   empty page 4\n```\n\n**Only per_page=100 loses data.** 99 is fine. 101 is fine. 100 comes up\n\nIt gets better. Those four pages of 100:\n\n```\nrows returned : 400\nunique ids    : 383\nids appearing on more than one page: 17\n  id 4267457 appears 2x on pages [2, 3]\n  id 4228478 appears 2x on pages [2, 3]\n  ...\n```\n\nFour pages of 100 returned exactly 400 rows — the count you would sanity-check against — while containing **383 articles, 17 of them twice, and 39 not at all**. The 39 are not random: the `per_page=100`\n\nwalk reaches back to 2026-07-14, the full listing reaches back to 2026-06-21. **It is the oldest entries that vanish**, which is precisely the population you would use to say anything about how the challenge has changed over time.\n\nI want to be careful about what I am claiming here. I did not find the cause inside DEV's code; I have no access to it. What I measured is that **the same query returns 422 or 383 items depending only on page size, and the short answer terminates cleanly.** That is reproducible from any machine, in about sixty requests, with no credentials.\n\n``` python\ndef assert_listing_stable(fetch, tag, sizes=(99, 100, 101, 200)):\n    \"\"\"A listing you cannot reproduce at two page sizes is not a population.\"\"\"\n    counts = {}\n    for pp in sizes:\n        ids, page = set(), 1\n        while True:\n            rows = fetch(tag, pp, page)\n            if not rows:\n                break\n            ids.update(r[\"id\"] for r in rows)\n            page += 1\n        counts[pp] = ids\n    best = max(counts.values(), key=len)\n    for pp, ids in counts.items():\n        if len(ids) != len(best):\n            raise RuntimeError(\n                f\"per_page={pp} yields {len(ids)} ids, per_page=\"\n                f\"{max(counts, key=lambda k: len(counts[k]))} yields {len(best)}\"\n            )\n    return best\n```\n\nNine lines, and it fails the build on an API defect I could not have guessed at.\n\nThe offline reproducer is self-contained and needs nothing but the standard library. It builds a corpus whose property-of-interest is correlated with position — which is the case that matters — and runs both collectors against a server that refuses in bursts.\n\n```\n\"\"\"A rate limiter that answers with the words \"Retry later\" is not an error your\nparser will notice. It is a sampler you did not know you installed.\n\nRuns offline against a fake server, so the numbers are reproducible without\ntouching anyone's API.\n\n  collect_naive()   drops anything that fails to parse and returns what it got\n  collect_strict()  treats \"did not parse\" as \"not fetched yet\", slows down,\n                    retries, and refuses to return until every id is accounted for\n\nRequires: nothing but the standard library.\n\"\"\"\nimport sys\n\nif hasattr(sys.stdout, \"reconfigure\"):\n    sys.stdout.reconfigure(encoding=\"utf-8\", errors=\"replace\")\n\nimport json\n\nN = 412              # items in the listing\nBURST = 8            # requests served before the limiter trips\nCOOLDOWN = 16        # requests refused before it serves again\n\ndef truth(i):\n    \"\"\"Ground truth: does item i belong to the category being counted?\n\n    The first third of the listing is denser than the rest -- newer submissions\n    mention the prize category more often than older ones. That is the part that\n    matters: the property being counted is correlated with position.\n    \"\"\"\n    threshold = 40 if i < N // 3 else 15\n    return (i * 37) % 100 < threshold\n\nclass Server:\n    \"\"\"Serves JSON in bursts. When the limiter trips it answers 'Retry later'.\n\n    The refusal is a 200 with a plain-text body. Nothing raises, nothing retries\n    itself, and the caller gets a str where it expected JSON.\n    \"\"\"\n\n    def __init__(self, cooldown=COOLDOWN):\n        self.n = 0\n        self.cooldown = cooldown\n        self.served = 0\n        self.refused = 0\n\n    def get(self, i):\n        self.n += 1\n        if self.cooldown and (self.n % (BURST + self.cooldown)) > BURST:\n            self.refused += 1\n            return \"Retry later\\n\"\n        self.served += 1\n        return json.dumps({\"id\": i, \"in_category\": truth(i)})\n\ndef parse(raw):\n    try:\n        return json.loads(raw)\n    except Exception:\n        return None\n\ndef collect_naive(server, ids):\n    \"\"\"What I actually wrote. There is no bug in it -- it does exactly what it says.\"\"\"\n    out = []\n    for i in ids:\n        rec = parse(server.get(i))\n        if rec is not None:\n            out.append(rec)\n    return out\n\ndef collect_strict(server, ids, max_rounds=8):\n    \"\"\"Same fetch, two extra rules: unparsed is not absent, and slow down on refusal.\"\"\"\n    done, pending, rounds = {}, list(ids), 0\n    while pending and rounds < max_rounds:\n        rounds += 1\n        still = []\n        for i in pending:\n            rec = parse(server.get(i))\n            if rec is None:\n                still.append(i)\n            else:\n                done[i] = rec\n        pending = still\n        server.cooldown //= 2          # back off: fewer requests per unit time\n    missing = [i for i in ids if i not in done]\n    if missing:\n        raise RuntimeError(f\"{len(missing)} of {len(ids)} ids never resolved\")\n    return [done[i] for i in ids], rounds\n\ndef share(records):\n    n = len(records)\n    hits = sum(1 for r in records if r[\"in_category\"])\n    return hits, n, (hits / n * 100 if n else 0.0)\n\ndef main():\n    ids = list(range(N))\n    actual = [{\"id\": i, \"in_category\": truth(i)} for i in ids]\n\n    naive = collect_naive(Server(), ids)\n    strict, rounds = collect_strict(Server(), ids)\n\n    print(f\"{'':<20}{'items':>8}{'coverage':>11}{'in category':>14}{'share':>9}\")\n    for label, recs in ((\"ground truth\", actual),\n                        (\"naive collector\", naive),\n                        (\"strict collector\", strict)):\n        hits, n, pct = share(recs)\n        print(f\"{label:<20}{n:>8}{n / N * 100:>10.1f}%{hits:>14}{pct:>8.1f}%\")\n\n    _, _, t = share(actual)\n    _, _, g = share(naive)\n    print(f\"\\nThe naive collector reported {g:.1f}% from {len(naive)}/{N} items. \"\n          f\"The answer is {t:.1f}%.\")\n    print(f\"It raised nothing, logged nothing, and its table looked complete.\")\n    print(f\"The strict collector resolved every id in {rounds} rounds.\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\nThe page-size sweep is fifteen lines of `curl`\n\naround the same idea and is quoted in full above.\n\n**Done:**\n\n`assert_listing_stable`\n\nruns before the fetch. On this tag it fails, correctly.**Done because the review said so:**\n\n**Not done, and I'd rather say so:**\n\n`curl -o`\n\nand `resp.raise_for_status()`\n\nwould have caught the whole thing is one I destroyed the evidence for. Sequential requests do not reproduce it, and I am not going to hammer a free API until it stops talking to me just to find out. The lesson stands on its own: **Left as a guard, deliberately:**\n\nRecomputing the denominator at two page sizes every time, even when nothing has changed. It costs about sixty requests and it is the only reason I know the number at all.\n\nI used **Gemini** (free tier, Flash) once: after the fix worked, with the numbers already in hand, to ask what my fix still could not see. I gave it both wrong answers, the loader, the strict collector, the assertion, and four questions. I did not adopt any of it — I measured each claim.\n\nIt named the class first:\n\nThis is a\n\nSilent Partial Sample(or Silent Truncation) resulting fromPlausible Degradation. [...] The system experienced soft failures—surrogate metrics (Attempt 1) and swallowed rate-limit payloads (Attempt 2)—that degraded the data quality into a plausible subset rather than triggering an explicit system error.\n\nThat one sentence covers both of my wrong answers, which is what I had asked for and had not managed to write myself. Then four ranked failure modes, and a verdict on the fix.\n\n```\nClaim                                             Verdict        Measurement\nRank 1  the listing itself is truncated           held           per_page=100 -> 383 of 422,\n                                                                 17 duplicated rows, empty page 5\nRank 2  valid JSON that lacks the key fields      held           the deleted entry parses fine and\n                                                                 has no body_markdown\nRank 4  HTTP 200 carrying an error payload        not observed   the deleted entry returns a real 404\nQ3      backoff+assert is a patch, not a fix      accepted       and see \"Not done\" above\nQ4      I may not call the 32.5% sample biased    conceded       claim retracted, see below\n```\n\n**Rank 1 is the reason this post exists.** I asked what my completeness check could not see, and the answer was: the thing it checks against. Sixty requests later I had an API defect that is reproducible by anyone, in which the page size everybody reaches for first is the only one that loses data.\n\n**Question 4 cost me a claim I liked.** I had written that the 32.5% sample was *biased*, not merely small, because rate-limit refusals arrive in bursts and bursts land on neighbours in a listing. Gemini pointed out that I overwrote the failed responses with the successful refetch, so I no longer have the evidence for that:\n\nYou cannot claim the 32.5% sample was definitively biased due to listing-order burst clustering. You cannot claim spatial, chronological, or network locality for the failed items, because you lost the exact temporal sequence and per-request metadata.\n\nWhat it left me is narrower and I think stronger:\n\nThe 32.5% sample is unvalidated and methodologically unreliable because the sampling mechanism was governed by server load-shedding rather than random selection. [...] the sample cannot be assumed to be Missing Completely at Random (MCAR).\n\nSo: 23.9% versus 18.5% is an empirical divergence of 5.4 points, and the sample was drawn by a server deciding what it felt like answering. That is enough to throw the number away. It is not enough to say *why* it leaned the way it did, and I have edited that claim out of my notes.\n\nOne last thing, which happened while I was measuring the above. My sweep script crashed decoding `curl`\n\n's output — the console here is cp932 and one of the titles was not — and my error handler, which was written to notice `Retry later`\n\n, reported it as **\"rate-limited at page 1\"** for all nine page sizes. I spent a minute believing the API had cut me off, in the middle of writing a post about mistaking a local failure for an absent record.\n\nThat is the whole thing, really. Six of my submissions this month are the same shape: *every check passed and the answer was still wrong.* A ranking only I could see. A collector reporting success on 27% of the data. A rate limiter counting the retries. A verifier that read every character back while three links in the document pointed at nothing. And now a denominator that was wrong three ways, where the third way was invisible until I asked something outside my own head what my check was not looking at.", "url": "https://wpnews.pro/news/i-paginated-by-100-and-lost-39-of-422-rows-at-99-and-101-nothing-was-missing", "canonical_source": "https://dev.to/aiq_labs/i-paginated-by-100-and-lost-39-of-422-rows-at-99-and-101-nothing-was-missing-hf", "published_at": "2026-08-22 02:15:13+00:00", "updated_at": "2026-08-22 02:43:24.818215+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["DEV", "Gemini", "Sentry"], "alternates": {"html": "https://wpnews.pro/news/i-paginated-by-100-and-lost-39-of-422-rows-at-99-and-101-nothing-was-missing", "markdown": "https://wpnews.pro/news/i-paginated-by-100-and-lost-39-of-422-rows-at-99-and-101-nothing-was-missing.md", "text": "https://wpnews.pro/news/i-paginated-by-100-and-lost-39-of-422-rows-at-99-and-101-nothing-was-missing.txt", "jsonld": "https://wpnews.pro/news/i-paginated-by-100-and-lost-39-of-422-rows-at-99-and-101-nothing-was-missing.jsonld"}}