# Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1% > Source: > Published: 2026-11-01 01:16:54+00:00 💡 **AI Labo Benchmark Series — Experiment `E16` (Ranked #3 out of 41 Patterns)** This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite. `E16` (Backlog Candidate `C012`)` e16-20261028-vm220-01` `VM220` (`ai-e16-220`, 0 NICs, `virtio-serial` QGA relay, `systemd` UID separation)`gemini-3.8-flash` (`thinkingLevel: LOW`)` rtk-ai/rtk`, Rust Token Killer) & SWE-agent Output Compression — What is the single biggest token sink in an autonomous coding agent loop? Surprisingly, it isn't the source code itself—it is **unfiltered CLI test runner output**. When an agent executes `pytest -vv --tb=long`, `cargo test`, or `npm test` on a realistic repository where 76 tests pass and 4 fail, dumping the raw terminal output (ANSI color codes, environment banners, 76 `[PASS]` lines, captured `DEBUG` stdout, and multi-frame internal tracebacks) directly into chat history adds **60,000+ characters (`28,000+` tokens) per test run**. To measure the impact of deterministic CLI output filtering, we benchmarked a 3-module payment settlement pipeline (`validator.py`, `settlement.py`, `statement.py`) across two harness modes inside isolated VM `VM220`: `RawTerminalOutputRunner`):` pytest` terminal stdout (`60,749 chars` on Turn 1; `117,391 chars` cumulative) directly to the conversation history.`TokenKillerFilterRunner`):` 1,506-character` structured JSON summary of the 4 failed assertions Compressing raw CLI test output from **`117,391 chars` ➔ `1,604 chars` (`-98.6%` reduction, `73.2x` compression)** enabled Mode B to fix all three modules in a single edit turn (`2` LLM calls total vs. `3` in Mode A). Mode B cut cumulative input tokens by **`-94.1%` (`92,385` ➔ `5,445` tokens, `16.97x` efficiency)**, API cost by **`-87.0%` (`$0.0770` ➔ `$0.0100`)**, and wall-clock time by **`-35.0%` (`1.54x` faster)** while achieving **`20 / 20` (100%)** on the hidden grading suite. | Metric | Mode A: Raw Terminal Output ( `RawTerminalOutput` ) | Mode B: Semantic JSON Filter ( `TokenKillerFilter` ) | Delta / Improvement | |---|---|---|---| | **Hidden Grading Suite Score (20 Cases)** | `20 / 20` (100%) | **`20 / 20` (100%)** | Both passed all 20 cases | | **Cumulative CLI Tool Output (`tool_output_chars`)** | `117,391 chars` | **`1,604 chars`** | **-98.6% (73.19x compression)** | | **First Test Run Output Size** | `60,749 chars` | **`1,506 chars`** | **-97.5% (76 PASS lines & ANSI stripped)** | | **LLM Calls (`model_calls`)** | `3 calls` | **`2 calls` (1 test run + 1 full fix)** | **-33.3% (No intermediate partial patch)** | | **Turn 1 Input / Output Tokens** | `2,406` in /`10` out | **`2,415` in / `10` out** | Identical initial test invocation | | **Turn 2 Input / Output Tokens** | `31,170` in /`484` out | **`3,030` in / `1,561` out** | **Turn 2 input cut by -90.3%** | | **Turn 3 Input / Output Tokens** | `58,809` in /`1,574` out | — *(Completed on Turn 2)* | **Avoided `58,809`-token log snowball** | | **Cumulative Input Tokens (`promptTokenCount`)** | `92,385 tokens` | **`5,445 tokens`** | **-94.1% (16.97x token efficiency)** | | **Cumulative Output Tokens (`candidatesTokenCount`)** | `2,068 tokens` | **`1,571 tokens`** | **-24.0% reduction** | | **Elapsed Execution Time (`elapsed_seconds`)** | `25.94 sec` | **`16.88 sec`** | **-35.0% (1.54x faster)** | | **Estimated API Cost (`estimated_micro_usd`)** | `$0.0770` (`77,045 micro-USD` ) | **`$0.0100` (`9,976 micro-USD`)** | **-87.0% (7.72x cheaper)** | ``` +-------------------------------------------------------------------------+ | Proxmox Isolated VM220 (0 NICs, virtio-serial QGA, systemd UID split) | | Target Suite: 80-Test Suite (76 PASS / 4 FAIL across 3 modules) | | | | +-------------------------------------------------------------------+ | | | Mode A: RawTerminalOutputRunner (Unfiltered pytest -vv --tb=long) | | | | - Turn 1: run_raw_pytest --> Returns 60,749 chars raw log | | | | - Turn 2: stage_partial_fix --> Input jumps to 31,170 tokens! | | | | Returns 56,544 chars raw log | | | | - Turn 3: submit_settlement --> Input hits 58,809 tokens! | | | | (Total: 92,385 input tokens) | | | +-------------------------------------------------------------------+ | | | | +-------------------------------------------------------------------+ | | | Mode B: TokenKillerFilterRunner (RTK-Style Semantic JSON Filter) | | | | - Turn 1: run_filtered_tests --> Strips 76 PASS & ANSI noise; | | | | returns 1,506-char failure JSON | | | | - Turn 2: submit_settlement --> Input is only 3,030 tokens! | | | | Fixes all 3 files in 1 shot | | | | (Total: 5,445 input tokens) | | | +-------------------------------------------------------------------+ | +-------------------------------------------------------------------------+ ``` `RawTerminalOutput`): How 2 Test Runs Burned 92,385 Input Tokens Tracking `execution-traces.json` and `usage.json` reveals how fast raw terminal output compounds in an agent's message array: `2,406 in / 10 out`):`{"tool": "run_raw_pytest"}`. The runner executes the 80-test suite and returns `60,749 characters``[PASS]` lines with `AUDIT_TRACE` dumps, and long multi-frame tracebacks for the 4 failures.`31,170 in / 484 out`):`+28,764 tokens`` 2,406` ➔ `31,170`). Distracted by the 60KB wall of text, the agent only patches `validator.py` (`stage_partial_fix`), triggering a second test run that appends another `56,544 characters`` 58,809 in / 1,574 out`):` 58,809 input tokens``92,385` cumulative input tokens).`TokenKillerFilter`): 1,506 Characters of Pure Signal When Mode B called `{"tool": "run_filtered_tests"}` on Turn 1, the harness preprocessor stripped all 76 passing tests, ANSI escape sequences, and `DEBUG` noise—returning a **`1,506-character` (`615-token`) JSON payload** containing only the 4 failing tests (` test`, `file`, `line`, `assertion`, `expected_vs_actual`): ``` { "summary": {"total": 80, "passed": 76, "failed": 4, "filtered_noise_chars_removed": 59243}, "failures": [ { "test": "test_validator_contract", "file": "validator.py", "line": 14, "assertion": "ALLOWED_CURRENCIES must include 'JPY'; reject empty list & bool; allow unit_cents==0 & chargeback_bps==1000; do not mutate input dict." }, { "test": "test_settlement_ordering_and_reserve", "file": "settlement.py", "line": 22, "assertion": "Preserve first-seen (merchant_id, currency) order (never sorted(order)); apply 10% reserve strictly when chargeback_bps > 300." }, { "test": "test_settlement_discounts_and_rebate_cap", "file": "settlement.py", "line": 39, "assertion": "Discount boundary is subtotal >= 15000 (GROWTH 10%, ENTERPRISE 20% else 10%); cap rebate_applied = min(fee_rebate, after_discount)." }, { "test": "test_statement_format_and_signed_zero", "file": "statement.py", "line": 11, "assertion": "Use '#' delimiter ({merchant_id}#{currency}:) and preserve leading '-' on zero discount/rebate (TIER_DISCOUNT: -0, REBATE_APPLIED: -0)." } ] } ``` Because all four failure root causes were immediately visible without scrolling through 76 passing tests, Mode B fixed all three files in a single Turn 2 call (`3,030` input tokens)—eliminating the intermediate rework turn entirely. `labjudge16`) Both modes passed all 20 hidden unit and E2E test cases verified by `labjudge16`: | # | Test Case ID | Invariant / Boundary Verified | Mode A ( `Raw CLI` ) | Mode B ( `Token Killer` ) | |---|---|---|---|---| | 1–6 | `validator_01` to`validator_06` | Trim/uppercase, non-mutation, `JPY` ,`bool` rejection,`0` cents &`1000` bps bounds | `6 / 6` PASS | **`6 / 6` PASS** | | 7–15 | `settlement_01` to`settlement_09` | First-seen order, conflict check, `300 vs 301` bps,`14999 vs 15000` tier, rebate cap | `9 / 9` PASS | **`9 / 9` PASS** | | 16–20 | `e2e_01` to`e2e_05` | `#` separator, signed zero`-0` ,`$0` auth statement, line order, error propagation | `5 / 5` PASS | **`5 / 5` PASS** | | **Total** | **20 Hidden Cases** | — | **`20 / 20` (100%)** | **`20 / 20` (100%)** | Never wire raw `subprocess.run(["pytest"])` stdout directly into your LLM agent's tool response. Instead: `pytest --tb=short -q --no-header --no-summary` (or `pytest-json-report`) inside your tool wrapper.`"passed": 500` as a single integer plus the exact file, line number, and assertion diff for the 2 failures.`rtk` (`Rust Token Killer`) or a 30-line Python wrapper around your test runner immediately cut agent token bills by `E20`):` MEMORY.md``E21`):