Why did my benchmark stop at N=22? A debugging story in nine bugs

A developer debugging the a2a-benchmark project discovered that a Python benchmark stopped at N=22 due to a CPython 3.11 integer-to-string conversion limit, which caused a silent crash. The fix revealed additional measurement errors, including a parser relying on LLM phrasing, a unit mismatch in Rust's time formatting, and a 400x performance gap between direct and LLM-mediated agents. The developer resolved nine bugs, including a case where Go's nanosecond precision broke the parser and another where Gemini's session history caused skipped runs.

Submission for DEV's Summer Bug Smash — Smash Stories track. There was a file in my repo called run benchmark 1 22.py . Not 1 to 24, which is what the harness was written to do. Not 1 to 26, which is how many Mersenne exponents the agents know. Twenty-two. A chart in the README — a2a latency times 1 22.png — agreed. At some point, past-me had decided the benchmark ends at 22, committed the evidence, and moved on. This summer, hunting for a Bug Smash target, I finally asked: why 22? a2a-benchmark https://github.com/xbill9/a2a-benchmark compares A2A agent performance across four languages. Python and Go sit behind Gemini tool-calling ADK ; Node and Rust are bare HTTP handlers. Each computes Mersenne primes with Lucas–Lehmer; a harness sweeps N from 1 to 24 and draws two charts. I ran the full sweep. At N=24, the Python column printed N/A . Every other language returned data. There it was — not a decision, a crash , worked around by shortening the run until it stopped hurting. The Python agent's response at N=24 wasn't even subtle about it: "Exceeds the limit 4300 digits for integer string conversion; use sys.set int max str digits to increase the limit" CPython 3.11 added a default cap on int→str conversion — 4,300 digits — as a denial-of-service mitigation. My agent stringified every prime it found. The 24th Mersenne prime, 2^19937−1, has 6,002 digits . Here's the part that made me laugh out loud: the stringified list was never returned . The tool reports only its elapsed time. The line that had silently amputated my benchmark at N=23 was decorative. The fix was git rm energy: delete the str , keep the raw int. Go had the identical dead weight val.String inside its timed region — it just happened not to crash. One deleted expression, and a column of data that had never existed came into being: N=24, Python, 2,425.9 ms. With the agents finally running, I kept pulling the thread. The harness parsed Python's elapsed time out of the LLM's prose with r"It took \d\.\-e + seconds" . Gemini, in my captures, never once said "It took" — it said "Calculating the first 5 Mersenne primes took…" and later "The calculation took…" . The only reason the benchmark had Python data at all was a fallback that read the structured tool artifact. My measurement pipeline's primary path was a bet on a language model's phrasing habits. The direct agents had their own tells. Ask Node or Rust for 100 Mersenne primes and they'd cheerfully report "Found first 100 Mersenne primes" — having computed 26, the size of their exponent table. And they formatted elapsed time as %.2f ms, so Rust's fastest runs reported 0.00ms , which parses to zero, which cannot exist on a log-scale chart. Those points didn't look wrong; they looked like nothing. And the biggest lie was the chart itself: "A2A Round-Trip Time including LLM/Tool calling ". Only half true — literally. Two of the four agents route through Gemini; two never touch an LLM. Median RTT: 2.6ms and 4.6ms for the direct pair, ~1.6s and ~1.8s for the Gemini pair. A ~400× gap presented as a language comparison was actually a pipeline comparison. I fixed everything and re-ran the sweep to generate the "after" charts. Go's N=1 datapoint: N/A . Cause: my fix. With the dead formatting deleted, Go's small-N runs got so fast that time.Duration switched output units — Elapsed time: 836ns — and the harness parser had branches for µs, ms, and s, but had never met a nanosecond. The fix made the code too fast for its own benchmark. Parser patched. Re-ran again. Three Go datapoints missing — different ones. The captured response text: "I already did that. Do you want to do it again?" The harness reused deterministic context IDs; ADK keeps per-context session history; on a rerun, Gemini looked at the old conversation and declined to redo the work. My benchmark's completeness now depended on a language model's opinions about repetition. Unique per-run IDs fixed it, and the final sweep came back 96/96 . run benchmark 1 22.py sat in the repo like a fossil of an uninvestigated crash. The moment you rename the script instead of reading the stack trace, you've decided to ship the bug.Nine bugs. Four PRs 1 https://github.com/xbill9/a2a-benchmark/pull/1 , 2 https://github.com/xbill9/a2a-benchmark/pull/2 , 3 https://github.com/xbill9/a2a-benchmark/pull/3 , 4 https://github.com/xbill9/a2a-benchmark/pull/4 . One question I should have asked a year ago: why 22? Disclosure: I ran this investigation with Claude Code as the debugging agent — it did the reproduction, the fixes, and the benchmark reruns while I steered. The bugs, the numbers, and the "I already did that" refusal are all real and archived in the repo.