# Why did my benchmark stop at N=22? A debugging story in nine bugs

> Source: <https://dev.to/xbill/why-did-my-benchmark-stop-at-n22-a-debugging-story-in-nine-bugs-3m2l>
> Published: 2026-07-15 18:56:45+00:00

*Submission for DEV's Summer Bug Smash — Smash Stories track.*

There was a file in my repo called `run_benchmark_1_22.py`

.

Not 1 to 24, which is what the harness was written to do. Not 1 to 26, which is how many Mersenne exponents the agents know. Twenty-two. A chart in the README — `a2a_latency_times_1_22.png`

— agreed. At some point, past-me had decided the benchmark ends at 22, committed the evidence, and moved on.

This summer, hunting for a Bug Smash target, I finally asked: *why 22?*

[ a2a-benchmark](https://github.com/xbill9/a2a-benchmark) compares A2A agent performance across four languages. Python and Go sit behind Gemini tool-calling (ADK); Node and Rust are bare HTTP handlers. Each computes Mersenne primes with Lucas–Lehmer; a harness sweeps N from 1 to 24 and draws two charts.

I ran the full sweep. At N=24, the Python column printed `N/A`

. Every other language returned data. There it was — not a decision, a **crash**, worked around by shortening the run until it stopped hurting.

The Python agent's response at N=24 wasn't even subtle about it:

```
"Exceeds the limit (4300 digits) for integer string conversion;
 use sys.set_int_max_str_digits() to increase the limit"
```

CPython 3.11 added a default cap on `int→str`

conversion — 4,300 digits — as a denial-of-service mitigation. My agent stringified every prime it found. The 24th Mersenne prime, 2^19937−1, has **6,002 digits**.

Here's the part that made me laugh out loud: the stringified list was *never returned*. The tool reports only its elapsed time. The line that had silently amputated my benchmark at N=23 was decorative. The fix was `git rm`

energy: delete the `str()`

, keep the raw int. Go had the identical dead weight (`val.String()`

) inside its timed region — it just happened not to crash.

One deleted expression, and a column of data that had never existed came into being: N=24, Python, 2,425.9 ms.

With the agents finally running, I kept pulling the thread. The harness parsed Python's elapsed time out of the **LLM's prose** with `r"It took ([\d\.\-e]+) seconds"`

. Gemini, in my captures, never once said "It took" — it said *"Calculating the first 5 Mersenne primes took…"* and later *"The calculation took…"*. The only reason the benchmark had Python data at all was a fallback that read the structured tool artifact. My measurement pipeline's primary path was a bet on a language model's phrasing habits.

The direct agents had their own tells. Ask Node or Rust for 100 Mersenne primes and they'd cheerfully report `"Found first 100 Mersenne primes"`

— having computed 26, the size of their exponent table. And they formatted elapsed time as `%.2f`

ms, so Rust's fastest runs reported `0.00ms`

, which parses to zero, which cannot exist on a log-scale chart. Those points didn't look wrong; they looked like nothing.

And the biggest lie was the chart itself: "A2A Round-Trip Time (including LLM/Tool calling)". Only half true — literally. Two of the four agents route through Gemini; two never touch an LLM. Median RTT: 2.6ms and 4.6ms for the direct pair, ~1.6s and ~1.8s for the Gemini pair. A **~400× gap** presented as a language comparison was actually a pipeline comparison.

I fixed everything and re-ran the sweep to generate the "after" charts. Go's N=1 datapoint: `N/A`

.

Cause: my fix. With the dead formatting deleted, Go's small-N runs got so fast that `time.Duration`

switched output units — `Elapsed time: 836ns`

— and the harness parser had branches for µs, ms, and s, but had never met a nanosecond. **The fix made the code too fast for its own benchmark.**

Parser patched. Re-ran again. Three Go datapoints missing — different ones. The captured response text:

"I already did that. Do you want to do it again?"

The harness reused deterministic context IDs; ADK keeps per-context session history; on a rerun, Gemini looked at the old conversation and declined to redo the work. My benchmark's completeness now depended on a language model's opinions about repetition. Unique per-run IDs fixed it, and the final sweep came back **96/96**.

`run_benchmark_1_22.py`

sat in the repo like a fossil of an uninvestigated crash. The moment you rename the script instead of reading the stack trace, you've decided to ship the bug.Nine bugs. Four PRs ([#1](https://github.com/xbill9/a2a-benchmark/pull/1), [#2](https://github.com/xbill9/a2a-benchmark/pull/2), [#3](https://github.com/xbill9/a2a-benchmark/pull/3), [#4](https://github.com/xbill9/a2a-benchmark/pull/4)). One question I should have asked a year ago: *why 22?*

*Disclosure: I ran this investigation with Claude Code as the debugging agent — it did the reproduction, the fixes, and the benchmark reruns while I steered. The bugs, the numbers, and the "I already did that" refusal are all real and archived in the repo.*