Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%

💡 AI Labo Benchmark Series — Experiment E16 (Ranked #3 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

E16 (Backlog Candidate C012) e16-20261028-vm220-01 VM220 (ai-e16-220, 0 NICs, virtio-serial QGA relay, systemd UID separation)gemini-3.8-flash (thinkingLevel: LOW) rtk-ai/rtk, Rust Token Killer) & SWE-agent Output Compression — What is the single biggest token sink in an autonomous coding agent loop? Surprisingly, it isn't the source code itself—it is unfiltered CLI test runner output.

When an agent executes pytest -vv --tb=long, cargo test, or npm test on a realistic repository where 76 tests pass and 4 fail, dumping the raw terminal output (ANSI color codes, environment banners, 76 [PASS] lines, captured DEBUG stdout, and multi-frame internal tracebacks) directly into chat history adds 60,000+ characters (28,000+ tokens) per test run.

To measure the impact of deterministic CLI output filtering, we benchmarked a 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) across two harness modes inside isolated VM VM220:

RawTerminalOutputRunner): pytest terminal stdout (60,749 chars on Turn 1; 117,391 chars cumulative) directly to the conversation history.TokenKillerFilterRunner): 1,506-character structured JSON summary of the 4 failed assertions Compressing raw CLI test output from 117,391 chars ➔ 1,604 chars (-98.6% reduction, 73.2x compression) enabled Mode B to fix all three modules in a single edit turn (2 LLM calls total vs. 3 in Mode A). Mode B cut cumulative input tokens by -94.1% (92,385 ➔ 5,445 tokens, 16.97x efficiency), API cost by -87.0% ($0.0770 ➔ $0.0100), and wall-clock time by -35.0% (1.54x faster) while achieving 20 / 20 (100%) on the hidden grading suite.

+-------------------------------------------------------------------------+
| Proxmox Isolated VM220 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Suite: 80-Test Suite (76 PASS / 4 FAIL across 3 modules)        |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: RawTerminalOutputRunner (Unfiltered pytest -vv --tb=long) |  |
|  |  - Turn 1: run_raw_pytest     --> Returns 60,749 chars raw log    |  |
|  |  - Turn 2: stage_partial_fix  --> Input jumps to 31,170 tokens!   |  |
|  |                                   Returns 56,544 chars raw log    |  |
|  |  - Turn 3: submit_settlement  --> Input hits 58,809 tokens!       |  |
|  |                                   (Total: 92,385 input tokens)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: TokenKillerFilterRunner (RTK-Style Semantic JSON Filter)  |  |
|  |  - Turn 1: run_filtered_tests --> Strips 76 PASS & ANSI noise;    |  |
|  |                                   returns 1,506-char failure JSON |  |
|  |  - Turn 2: submit_settlement  --> Input is only 3,030 tokens!     |  |
|  |                                   Fixes all 3 files in 1 shot     |  |
|  |                                   (Total: 5,445 input tokens)     |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

RawTerminalOutput): How 2 Test Runs Burned 92,385 Input Tokens Tracking execution-traces.json and usage.json reveals how fast raw terminal output compounds in an agent's message array:

2,406 in / 10 out):{"tool": "run_raw_pytest"}. The runner executes the 80-test suite and returns 60,749 characters``[PASS] lines with AUDIT_TRACE dumps, and long multi-frame tracebacks for the 4 failures.31,170 in / 484 out):+28,764 tokens`` 2,406 ➔ 31,170). Distracted by the 60KB wall of text, the agent only patches validator.py (stage_partial_fix), triggering a second test run that appends another 56,544 characters`` 58,809 in / 1,574 out): 58,809 input tokens``92,385 cumulative input tokens).TokenKillerFilter): 1,506 Characters of Pure Signal When Mode B called {"tool": "run_filtered_tests"} on Turn 1, the harness preprocessor stripped all 76 passing tests, ANSI escape sequences, and DEBUG noise—returning a 1,506-character (615-token) JSON payload containing only the 4 failing tests ( test, file, line, assertion, expected_vs_actual):

{
  "summary": {"total": 80, "passed": 76, "failed": 4, "filtered_noise_chars_removed": 59243},
  "failures": [
    {
      "test": "test_validator_contract",
      "file": "validator.py",
      "line": 14,
      "assertion": "ALLOWED_CURRENCIES must include 'JPY'; reject empty list & bool; allow unit_cents==0 & chargeback_bps==1000; do not mutate input dict."
    },
    {
      "test": "test_settlement_ordering_and_reserve",
      "file": "settlement.py",
      "line": 22,
      "assertion": "Preserve first-seen (merchant_id, currency) order (never sorted(order)); apply 10% reserve strictly when chargeback_bps > 300."
    },
    {
      "test": "test_settlement_discounts_and_rebate_cap",
      "file": "settlement.py",
      "line": 39,
      "assertion": "Discount boundary is subtotal >= 15000 (GROWTH 10%, ENTERPRISE 20% else 10%); cap rebate_applied = min(fee_rebate, after_discount)."
    },
    {
      "test": "test_statement_format_and_signed_zero",
      "file": "statement.py",
      "line": 11,
      "assertion": "Use '#' delimiter ({merchant_id}#{currency}:) and preserve leading '-' on zero discount/rebate (TIER_DISCOUNT: -0, REBATE_APPLIED: -0)."
    }
  ]
}

Because all four failure root causes were immediately visible without scrolling through 76 passing tests, Mode B fixed all three files in a single Turn 2 call (3,030 input tokens)—eliminating the intermediate rework turn entirely.

labjudge16) Both modes passed all 20 hidden unit and E2E test cases verified by labjudge16:

Never wire raw subprocess.run(["pytest"]) stdout directly into your LLM agent's tool response. Instead:

pytest --tb=short -q --no-header --no-summary (or pytest-json-report) inside your tool wrapper."passed": 500 as a single integer plus the exact file, line number, and assertion diff for the 2 failures.rtk (Rust Token Killer) or a 30-line Python wrapper around your test runner immediately cut agent token bills by E20): MEMORY.md``E21):