{"slug": "why-did-my-benchmark-stop-at-n-22-a-debugging-story-in-nine-bugs", "title": "Why did my benchmark stop at N=22? A debugging story in nine bugs", "summary": "A developer debugging the a2a-benchmark project discovered that a Python benchmark stopped at N=22 due to a CPython 3.11 integer-to-string conversion limit, which caused a silent crash. The fix revealed additional measurement errors, including a parser relying on LLM phrasing, a unit mismatch in Rust's time formatting, and a 400x performance gap between direct and LLM-mediated agents. The developer resolved nine bugs, including a case where Go's nanosecond precision broke the parser and another where Gemini's session history caused skipped runs.", "body_md": "*Submission for DEV's Summer Bug Smash — Smash Stories track.*\n\nThere was a file in my repo called `run_benchmark_1_22.py`\n\n.\n\nNot 1 to 24, which is what the harness was written to do. Not 1 to 26, which is how many Mersenne exponents the agents know. Twenty-two. A chart in the README — `a2a_latency_times_1_22.png`\n\n— agreed. At some point, past-me had decided the benchmark ends at 22, committed the evidence, and moved on.\n\nThis summer, hunting for a Bug Smash target, I finally asked: *why 22?*\n\n[ a2a-benchmark](https://github.com/xbill9/a2a-benchmark) compares A2A agent performance across four languages. Python and Go sit behind Gemini tool-calling (ADK); Node and Rust are bare HTTP handlers. Each computes Mersenne primes with Lucas–Lehmer; a harness sweeps N from 1 to 24 and draws two charts.\n\nI ran the full sweep. At N=24, the Python column printed `N/A`\n\n. Every other language returned data. There it was — not a decision, a **crash**, worked around by shortening the run until it stopped hurting.\n\nThe Python agent's response at N=24 wasn't even subtle about it:\n\n```\n\"Exceeds the limit (4300 digits) for integer string conversion;\n use sys.set_int_max_str_digits() to increase the limit\"\n```\n\nCPython 3.11 added a default cap on `int→str`\n\nconversion — 4,300 digits — as a denial-of-service mitigation. My agent stringified every prime it found. The 24th Mersenne prime, 2^19937−1, has **6,002 digits**.\n\nHere's the part that made me laugh out loud: the stringified list was *never returned*. The tool reports only its elapsed time. The line that had silently amputated my benchmark at N=23 was decorative. The fix was `git rm`\n\nenergy: delete the `str()`\n\n, keep the raw int. Go had the identical dead weight (`val.String()`\n\n) inside its timed region — it just happened not to crash.\n\nOne deleted expression, and a column of data that had never existed came into being: N=24, Python, 2,425.9 ms.\n\nWith the agents finally running, I kept pulling the thread. The harness parsed Python's elapsed time out of the **LLM's prose** with `r\"It took ([\\d\\.\\-e]+) seconds\"`\n\n. Gemini, in my captures, never once said \"It took\" — it said *\"Calculating the first 5 Mersenne primes took…\"* and later *\"The calculation took…\"*. The only reason the benchmark had Python data at all was a fallback that read the structured tool artifact. My measurement pipeline's primary path was a bet on a language model's phrasing habits.\n\nThe direct agents had their own tells. Ask Node or Rust for 100 Mersenne primes and they'd cheerfully report `\"Found first 100 Mersenne primes\"`\n\n— having computed 26, the size of their exponent table. And they formatted elapsed time as `%.2f`\n\nms, so Rust's fastest runs reported `0.00ms`\n\n, which parses to zero, which cannot exist on a log-scale chart. Those points didn't look wrong; they looked like nothing.\n\nAnd the biggest lie was the chart itself: \"A2A Round-Trip Time (including LLM/Tool calling)\". Only half true — literally. Two of the four agents route through Gemini; two never touch an LLM. Median RTT: 2.6ms and 4.6ms for the direct pair, ~1.6s and ~1.8s for the Gemini pair. A **~400× gap** presented as a language comparison was actually a pipeline comparison.\n\nI fixed everything and re-ran the sweep to generate the \"after\" charts. Go's N=1 datapoint: `N/A`\n\n.\n\nCause: my fix. With the dead formatting deleted, Go's small-N runs got so fast that `time.Duration`\n\nswitched output units — `Elapsed time: 836ns`\n\n— and the harness parser had branches for µs, ms, and s, but had never met a nanosecond. **The fix made the code too fast for its own benchmark.**\n\nParser patched. Re-ran again. Three Go datapoints missing — different ones. The captured response text:\n\n\"I already did that. Do you want to do it again?\"\n\nThe harness reused deterministic context IDs; ADK keeps per-context session history; on a rerun, Gemini looked at the old conversation and declined to redo the work. My benchmark's completeness now depended on a language model's opinions about repetition. Unique per-run IDs fixed it, and the final sweep came back **96/96**.\n\n`run_benchmark_1_22.py`\n\nsat in the repo like a fossil of an uninvestigated crash. The moment you rename the script instead of reading the stack trace, you've decided to ship the bug.Nine bugs. Four PRs ([#1](https://github.com/xbill9/a2a-benchmark/pull/1), [#2](https://github.com/xbill9/a2a-benchmark/pull/2), [#3](https://github.com/xbill9/a2a-benchmark/pull/3), [#4](https://github.com/xbill9/a2a-benchmark/pull/4)). One question I should have asked a year ago: *why 22?*\n\n*Disclosure: I ran this investigation with Claude Code as the debugging agent — it did the reproduction, the fixes, and the benchmark reruns while I steered. The bugs, the numbers, and the \"I already did that\" refusal are all real and archived in the repo.*", "url": "https://wpnews.pro/news/why-did-my-benchmark-stop-at-n-22-a-debugging-story-in-nine-bugs", "canonical_source": "https://dev.to/xbill/why-did-my-benchmark-stop-at-n22-a-debugging-story-in-nine-bugs-3m2l", "published_at": "2026-07-15 18:56:45+00:00", "updated_at": "2026-07-15 19:13:31.868938+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models", "ai-agents"], "entities": ["a2a-benchmark", "Gemini", "Python", "Go", "Rust", "Node", "CPython", "ADK"], "alternates": {"html": "https://wpnews.pro/news/why-did-my-benchmark-stop-at-n-22-a-debugging-story-in-nine-bugs", "markdown": "https://wpnews.pro/news/why-did-my-benchmark-stop-at-n-22-a-debugging-story-in-nine-bugs.md", "text": "https://wpnews.pro/news/why-did-my-benchmark-stop-at-n-22-a-debugging-story-in-nine-bugs.txt", "jsonld": "https://wpnews.pro/news/why-did-my-benchmark-stop-at-n-22-a-debugging-story-in-nine-bugs.jsonld"}}