{"slug": "my-benchmark-s-python-column-was-n-a-for-a-year-cpython-s-4300-digit-limit-and", "title": "My benchmark's Python column was N/A for a year — CPython's 4300-digit limit, and eight other bugs", "summary": "A developer discovered that their multi-language A2A benchmark suite had a Python column showing N/A for a year due to CPython's 4300-digit integer-to-string conversion limit. The bug caused the Python agent to crash when computing Mersenne primes beyond the 24th exponent, as the stringified prime exceeded the limit. Additional issues included reliance on LLM prose for timing data, incorrect count reporting, and parser failures for sub-microsecond durations.", "body_md": "*Submission for DEV's Summer Bug Smash — Clear the Lineup track.*\n\n[ a2a-benchmark](https://github.com/xbill9/a2a-benchmark) is my multi-language A2A (Agent-to-Agent) performance suite: four agents — Python and Go behind Gemini tool-calling via ADK, Node.js and Rust as direct handlers — each compute Mersenne primes with the Lucas–Lehmer test, while a harness sweeps N=1–24 and charts calculation time and round-trip time.\n\nThe committed results stopped at N=22. I never questioned that. I should have.\n\nCPython 3.11+ [limits int→str conversion to 4,300 digits](https://docs.python.org/3/library/stdtypes.html#int-max-str-digits) by default (a DoS mitigation). My Python agent stringified each prime it found:\n\n```\nmersenne_primes.append(str((1 << p) - 1))\n```\n\nThe 24th Mersenne exponent is p=19937, and 2^19937−1 has **6,002 digits**. So for any request of 24+ primes, the tool raised `ValueError`\n\n— and the A2A response dutifully delivered the stack-trace text instead of a result:\n\n```\n\"text\": \"Exceeds the limit (4300 digits) for integer string conversion;\n         use sys.set_int_max_str_digits() to increase the limit\"\n```\n\nThe benchmark's Python column was structurally incapable of producing data at N≥24. The kicker: **the stringified list was never used**. The tool returns only `elapsed_time`\n\n. The fix is deleting the `str()`\n\n— which also removes formatting work from the timed region that the Node and Rust agents never paid (Go had the same dead `val.String()`\n\ncall).\n\n**Fix: PR #1** — plus a switch from\n\n`time.time()`\n\n(wall clock, non-monotonic) to `time.perf_counter()`\n\n, and a regression test at count=24.Before/after, N=24 row:\n\n| Node.js | Rust | Go | Python | |\n|---|---|---|---|---|\n| before | 1633.01 ms | 812.57 ms | 1451.49 ms | N/A (crash) |\n| after | 1616.13 ms | 824.10 ms | 1531.10 ms | 2425.9 ms |\n\nThe Python agent's timing came back in two places: a structured `elapsed_time`\n\nin the tool artifact, and the model's prose. The harness regexed the prose *first*:\n\n```\nm = re.search(r\"It took ([\\d\\.\\-e]+) seconds\", text)\n```\n\nIn live runs, Gemini said *\"Calculating the first 5 Mersenne primes took 4.9591064453125e-05 seconds.\"* one time and *\"The calculation took 2.40715261301375 seconds.\"* the next. Neither matches `\"It took\"`\n\n. Every datapoint survived only because a fallback happened to dig out the structured value. Measurement data should never depend on how an LLM felt like phrasing a sentence.\n\nMeanwhile Node and Rust reported `elapsed.toFixed(2)`\n\nms — so sub-10µs runs returned `0.00ms`\n\n, which parses to `0.0`\n\nand silently vanishes from a log-scale chart.\n\n**Fix: PR #2** — structured artifact first, prose as last resort; 4-decimal timing output.\n\nThe exponent table has 26 entries, but both direct agents echoed the *requested* count:\n\n``` bash\n$ curl ... \"Calculate the first 100 Mersenne primes\"\nnode: \"Found first 100 Mersenne primes in 4450.78ms.\"   # computed 26\nrust: \"Found first 100 Mersenne primes in 2160.45ms.\"   # computed 26\n```\n\n**Fix: PR #3** — report\n\n`primes.length`\n\n/ `primes.len()`\n\n.Python and Go requests route through Gemini tool-calling; Node and Rust regex the number out of the message and compute directly. The RTT chart presented all four as one comparison \"including LLM/Tool calling\" — true for exactly half the lines. Median RTT: **Rust 2.6ms, Node 4.6ms vs Go ~1.6s, Python ~1.8s**. That ~400× gap is pipeline architecture, not language performance.\n\n**Fix: PR #4** — agents tagged by pipeline; the chart renders\n\n**The fix made the code too fast for its own benchmark.** With formatting removed from Go's timed region, small-N runs dropped under a microsecond — and Go's `time.Duration`\n\nstarted printing `836ns`\n\n, a suffix the harness parser had never seen. The datapoint silently became N/A. One more parser branch fixed it.\n\n**Gemini refused to repeat itself.** My harness reused deterministic `contextId`\n\ns, and ADK keeps per-context session history. On a rerun, the model answered *\"I already did that. Do you want to do it again?\"* — without calling the tool. Three datapoints gone. Benchmark IDs are now unique per run.\n\n**96/96 datapoints**, up from a baseline where one language column crashed, small-N values were unplottable zeros, and two datapoints per rerun depended on an LLM's patience. Nine bugs, four PRs, one afternoon of reproduction scripts — all archived with before/after charts.\n\nThe suite runs on ADK + `gemini-2.5-flash`\n\ntool-calling end to end. Two hard-won lessons for anyone building Gemini agents: **read structured tool artifacts, never model prose, when a machine consumes the output** — and **never reuse context IDs for independent requests** unless you want session memory changing your results.\n\n*Disclosure: I used Claude Code as the debugging/automation agent for this work — it reproduced each bug, wrote the fixes and regression tests, and ran the before/after benchmark sweeps. Every bug, number, and PR above is real and verifiable in the repo.*", "url": "https://wpnews.pro/news/my-benchmark-s-python-column-was-n-a-for-a-year-cpython-s-4300-digit-limit-and", "canonical_source": "https://dev.to/xbill/my-benchmarks-python-column-was-na-for-a-year-cpythons-4300-digit-limit-and-eight-other-bugs-1hgk", "published_at": "2026-07-15 18:56:09+00:00", "updated_at": "2026-07-15 19:13:37.976957+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "ai-agents"], "entities": ["CPython", "Gemini", "ADK", "Node.js", "Rust", "Go", "Python"], "alternates": {"html": "https://wpnews.pro/news/my-benchmark-s-python-column-was-n-a-for-a-year-cpython-s-4300-digit-limit-and", "markdown": "https://wpnews.pro/news/my-benchmark-s-python-column-was-n-a-for-a-year-cpython-s-4300-digit-limit-and.md", "text": "https://wpnews.pro/news/my-benchmark-s-python-column-was-n-a-for-a-year-cpython-s-4300-digit-limit-and.txt", "jsonld": "https://wpnews.pro/news/my-benchmark-s-python-column-was-n-a-for-a-year-cpython-s-4300-digit-limit-and.jsonld"}}