Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%

An AI Labo benchmark experiment (E16) found that filtering raw CLI test-runner output into a structured JSON summary before feeding it to a coding agent cut cumulative input tokens by 94.1% (92,385 to 5,445) and API cost by 87.0% ($0.0770 to $0.0100). Running inside an isolated Proxmox VM against a hidden 20-case grading suite, the JSON filter harness compressed 117,391 characters of pytest output to 1,604 characters (-98.6%), fixed all three payment-pipeline modules in two LLM calls instead of three, and finished 35% faster while both modes scored 20/20.

💡 AI Labo Benchmark Series — Experiment E16 Ranked 3 out of 41 Patterns This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite. E16 Backlog Candidate C012 e16-20261028-vm220-01 VM220 ai-e16-220 , 0 NICs, virtio-serial QGA relay, systemd UID separation gemini-3.8-flash thinkingLevel: LOW rtk-ai/rtk , Rust Token Killer & SWE-agent Output Compression — What is the single biggest token sink in an autonomous coding agent loop? Surprisingly, it isn't the source code itself—it is unfiltered CLI test runner output . When an agent executes pytest -vv --tb=long , cargo test , or npm test on a realistic repository where 76 tests pass and 4 fail, dumping the raw terminal output ANSI color codes, environment banners, 76 PASS lines, captured DEBUG stdout, and multi-frame internal tracebacks directly into chat history adds 60,000+ characters 28,000+ tokens per test run . To measure the impact of deterministic CLI output filtering, we benchmarked a 3-module payment settlement pipeline validator.py , settlement.py , statement.py across two harness modes inside isolated VM VM220 : RawTerminalOutputRunner : pytest terminal stdout 60,749 chars on Turn 1; 117,391 chars cumulative directly to the conversation history. TokenKillerFilterRunner : 1,506-character structured JSON summary of the 4 failed assertions Compressing raw CLI test output from 117,391 chars ➔ 1,604 chars -98.6% reduction, 73.2x compression enabled Mode B to fix all three modules in a single edit turn 2 LLM calls total vs. 3 in Mode A . Mode B cut cumulative input tokens by -94.1% 92,385 ➔ 5,445 tokens, 16.97x efficiency , API cost by -87.0% $0.0770 ➔ $0.0100 , and wall-clock time by -35.0% 1.54x faster while achieving 20 / 20 100% on the hidden grading suite. | Metric | Mode A: Raw Terminal Output RawTerminalOutput | Mode B: Semantic JSON Filter TokenKillerFilter | Delta / Improvement | |---|---|---|---| | Hidden Grading Suite Score 20 Cases | 20 / 20 100% | 20 / 20 100% | Both passed all 20 cases | | Cumulative CLI Tool Output tool output chars | 117,391 chars | 1,604 chars | -98.6% 73.19x compression | | First Test Run Output Size | 60,749 chars | 1,506 chars | -97.5% 76 PASS lines & ANSI stripped | | LLM Calls model calls | 3 calls | 2 calls 1 test run + 1 full fix | -33.3% No intermediate partial patch | | Turn 1 Input / Output Tokens | 2,406 in / 10 out | 2,415 in / 10 out | Identical initial test invocation | | Turn 2 Input / Output Tokens | 31,170 in / 484 out | 3,030 in / 1,561 out | Turn 2 input cut by -90.3% | | Turn 3 Input / Output Tokens | 58,809 in / 1,574 out | — Completed on Turn 2 | Avoided 58,809 -token log snowball | | Cumulative Input Tokens promptTokenCount | 92,385 tokens | 5,445 tokens | -94.1% 16.97x token efficiency | | Cumulative Output Tokens candidatesTokenCount | 2,068 tokens | 1,571 tokens | -24.0% reduction | | Elapsed Execution Time elapsed seconds | 25.94 sec | 16.88 sec | -35.0% 1.54x faster | | Estimated API Cost estimated micro usd | $0.0770 77,045 micro-USD | $0.0100 9,976 micro-USD | -87.0% 7.72x cheaper | +-------------------------------------------------------------------------+ | Proxmox Isolated VM220 0 NICs, virtio-serial QGA, systemd UID split | | Target Suite: 80-Test Suite 76 PASS / 4 FAIL across 3 modules | | | | +-------------------------------------------------------------------+ | | | Mode A: RawTerminalOutputRunner Unfiltered pytest -vv --tb=long | | | | - Turn 1: run raw pytest -- Returns 60,749 chars raw log | | | | - Turn 2: stage partial fix -- Input jumps to 31,170 tokens | | | | Returns 56,544 chars raw log | | | | - Turn 3: submit settlement -- Input hits 58,809 tokens | | | | Total: 92,385 input tokens | | | +-------------------------------------------------------------------+ | | | | +-------------------------------------------------------------------+ | | | Mode B: TokenKillerFilterRunner RTK-Style Semantic JSON Filter | | | | - Turn 1: run filtered tests -- Strips 76 PASS & ANSI noise; | | | | returns 1,506-char failure JSON | | | | - Turn 2: submit settlement -- Input is only 3,030 tokens | | | | Fixes all 3 files in 1 shot | | | | Total: 5,445 input tokens | | | +-------------------------------------------------------------------+ | +-------------------------------------------------------------------------+ RawTerminalOutput : How 2 Test Runs Burned 92,385 Input Tokens Tracking execution-traces.json and usage.json reveals how fast raw terminal output compounds in an agent's message array: 2,406 in / 10 out : {"tool": "run raw pytest"} . The runner executes the 80-test suite and returns 60,749 characters PASS lines with AUDIT TRACE dumps, and long multi-frame tracebacks for the 4 failures. 31,170 in / 484 out : +28,764 tokens 2,406 ➔ 31,170 . Distracted by the 60KB wall of text, the agent only patches validator.py stage partial fix , triggering a second test run that appends another 56,544 characters 58,809 in / 1,574 out : 58,809 input tokens 92,385 cumulative input tokens . TokenKillerFilter : 1,506 Characters of Pure Signal When Mode B called {"tool": "run filtered tests"} on Turn 1, the harness preprocessor stripped all 76 passing tests, ANSI escape sequences, and DEBUG noise—returning a 1,506-character 615-token JSON payload containing only the 4 failing tests test , file , line , assertion , expected vs actual : { "summary": {"total": 80, "passed": 76, "failed": 4, "filtered noise chars removed": 59243}, "failures": { "test": "test validator contract", "file": "validator.py", "line": 14, "assertion": "ALLOWED CURRENCIES must include 'JPY'; reject empty list & bool; allow unit cents==0 & chargeback bps==1000; do not mutate input dict." }, { "test": "test settlement ordering and reserve", "file": "settlement.py", "line": 22, "assertion": "Preserve first-seen merchant id, currency order never sorted order ; apply 10% reserve strictly when chargeback bps 300." }, { "test": "test settlement discounts and rebate cap", "file": "settlement.py", "line": 39, "assertion": "Discount boundary is subtotal = 15000 GROWTH 10%, ENTERPRISE 20% else 10% ; cap rebate applied = min fee rebate, after discount ." }, { "test": "test statement format and signed zero", "file": "statement.py", "line": 11, "assertion": "Use ' ' delimiter {merchant id} {currency}: and preserve leading '-' on zero discount/rebate TIER DISCOUNT: -0, REBATE APPLIED: -0 ." } } Because all four failure root causes were immediately visible without scrolling through 76 passing tests, Mode B fixed all three files in a single Turn 2 call 3,030 input tokens —eliminating the intermediate rework turn entirely. labjudge16 Both modes passed all 20 hidden unit and E2E test cases verified by labjudge16 : | | Test Case ID | Invariant / Boundary Verified | Mode A Raw CLI | Mode B Token Killer | |---|---|---|---|---| | 1–6 | validator 01 to validator 06 | Trim/uppercase, non-mutation, JPY , bool rejection, 0 cents & 1000 bps bounds | 6 / 6 PASS | 6 / 6 PASS | | 7–15 | settlement 01 to settlement 09 | First-seen order, conflict check, 300 vs 301 bps, 14999 vs 15000 tier, rebate cap | 9 / 9 PASS | 9 / 9 PASS | | 16–20 | e2e 01 to e2e 05 | separator, signed zero -0 , $0 auth statement, line order, error propagation | 5 / 5 PASS | 5 / 5 PASS | | Total | 20 Hidden Cases | — | 20 / 20 100% | 20 / 20 100% | Never wire raw subprocess.run "pytest" stdout directly into your LLM agent's tool response. Instead: pytest --tb=short -q --no-header --no-summary or pytest-json-report inside your tool wrapper. "passed": 500 as a single integer plus the exact file, line number, and assertion diff for the 2 failures. rtk Rust Token Killer or a 30-line Python wrapper around your test runner immediately cut agent token bills by E20 : MEMORY.md E21 :