{"slug": "stop-dumping-raw-pytest-logs-into-your-coding-agent-how-a-structured-json-test-1", "title": "Stop Dumping Raw Pytest Logs Into Your Coding Agent: How a Structured JSON Test Filter Cuts Input Tokens by 94.1%", "summary": "An AI Labo benchmark experiment (E16) found that filtering raw CLI test-runner output into a structured JSON summary before feeding it to a coding agent cut cumulative input tokens by 94.1% (92,385 to 5,445) and API cost by 87.0% ($0.0770 to $0.0100). Running inside an isolated Proxmox VM against a hidden 20-case grading suite, the JSON filter harness compressed 117,391 characters of pytest output to 1,604 characters (-98.6%), fixed all three payment-pipeline modules in two LLM calls instead of three, and finished 35% faster while both modes scored 20/20.", "body_md": "💡 **AI Labo Benchmark Series — Experiment `E16` (Ranked #3 out of 41 Patterns)**\n\nThis post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.\n\n`E16` (Backlog Candidate `C012`)` e16-20261028-vm220-01`\n`VM220` (`ai-e16-220`, 0 NICs, `virtio-serial` QGA relay, `systemd` UID separation)`gemini-3.8-flash` (`thinkingLevel: LOW`)` rtk-ai/rtk`, Rust Token Killer) & SWE-agent Output Compression — What is the single biggest token sink in an autonomous coding agent loop? Surprisingly, it isn't the source code itself—it is **unfiltered CLI test runner output**.\n\nWhen an agent executes `pytest -vv --tb=long`, `cargo test`, or `npm test` on a realistic repository where 76 tests pass and 4 fail, dumping the raw terminal output (ANSI color codes, environment banners, 76 `[PASS]` lines, captured `DEBUG` stdout, and multi-frame internal tracebacks) directly into chat history adds **60,000+ characters (`28,000+` tokens) per test run**.\n\nTo measure the impact of deterministic CLI output filtering, we benchmarked a 3-module payment settlement pipeline (`validator.py`, `settlement.py`, `statement.py`) across two harness modes inside isolated VM `VM220`:\n\n`RawTerminalOutputRunner`):` pytest` terminal stdout (`60,749 chars` on Turn 1; `117,391 chars` cumulative) directly to the conversation history.`TokenKillerFilterRunner`):` 1,506-character` structured JSON summary of the 4 failed assertions\nCompressing raw CLI test output from **`117,391 chars` ➔ `1,604 chars` (`-98.6%` reduction, `73.2x` compression)** enabled Mode B to fix all three modules in a single edit turn (`2` LLM calls total vs. `3` in Mode A). Mode B cut cumulative input tokens by **`-94.1%` (`92,385` ➔ `5,445` tokens, `16.97x` efficiency)**, API cost by **`-87.0%` (`$0.0770` ➔ `$0.0100`)**, and wall-clock time by **`-35.0%` (`1.54x` faster)** while achieving **`20 / 20` (100%)** on the hidden grading suite.\n\n| Metric | Mode A: Raw Terminal Output ( `RawTerminalOutput` ) | Mode B: Semantic JSON Filter ( `TokenKillerFilter` ) | Delta / Improvement | \n|---|---|---|---|\n| **Hidden Grading Suite Score (20 Cases)** | `20 / 20` (100%) | **`20 / 20` (100%)** | Both passed all 20 cases | \n| **Cumulative CLI Tool Output (`tool_output_chars`)** | `117,391 chars` | **`1,604 chars`** | **-98.6% (73.19x compression)** | \n| **First Test Run Output Size** | `60,749 chars` | **`1,506 chars`** | **-97.5% (76 PASS lines & ANSI stripped)** | \n| **LLM Calls (`model_calls`)** | `3 calls` | **`2 calls` (1 test run + 1 full fix)** | **-33.3% (No intermediate partial patch)** | \n| **Turn 1 Input / Output Tokens** | `2,406` in /`10` out | **`2,415` in / `10` out** | Identical initial test invocation | \n| **Turn 2 Input / Output Tokens** | `31,170` in /`484` out | **`3,030` in / `1,561` out** | **Turn 2 input cut by -90.3%** | \n| **Turn 3 Input / Output Tokens** | `58,809` in /`1,574` out | — *(Completed on Turn 2)* | **Avoided `58,809`-token log snowball** | \n| **Cumulative Input Tokens (`promptTokenCount`)** | `92,385 tokens` | **`5,445 tokens`** | **-94.1% (16.97x token efficiency)** | \n| **Cumulative Output Tokens (`candidatesTokenCount`)** | `2,068 tokens` | **`1,571 tokens`** | **-24.0% reduction** | \n| **Elapsed Execution Time (`elapsed_seconds`)** | `25.94 sec` | **`16.88 sec`** | **-35.0% (1.54x faster)** | \n| **Estimated API Cost (`estimated_micro_usd`)** | `$0.0770` (`77,045 micro-USD` ) | **`$0.0100` (`9,976 micro-USD`)** | **-87.0% (7.72x cheaper)** | \n\n```\n+-------------------------------------------------------------------------+\n| Proxmox Isolated VM220 (0 NICs, virtio-serial QGA, systemd UID split)   |\n|  Target Suite: 80-Test Suite (76 PASS / 4 FAIL across 3 modules)        |\n|                                                                         |\n|  +-------------------------------------------------------------------+  |\n|  | Mode A: RawTerminalOutputRunner (Unfiltered pytest -vv --tb=long) |  |\n|  |  - Turn 1: run_raw_pytest     --> Returns 60,749 chars raw log    |  |\n|  |  - Turn 2: stage_partial_fix  --> Input jumps to 31,170 tokens!   |  |\n|  |                                   Returns 56,544 chars raw log    |  |\n|  |  - Turn 3: submit_settlement  --> Input hits 58,809 tokens!       |  |\n|  |                                   (Total: 92,385 input tokens)    |  |\n|  +-------------------------------------------------------------------+  |\n|                                                                         |\n|  +-------------------------------------------------------------------+  |\n|  | Mode B: TokenKillerFilterRunner (RTK-Style Semantic JSON Filter)  |  |\n|  |  - Turn 1: run_filtered_tests --> Strips 76 PASS & ANSI noise;    |  |\n|  |                                   returns 1,506-char failure JSON |  |\n|  |  - Turn 2: submit_settlement  --> Input is only 3,030 tokens!     |  |\n|  |                                   Fixes all 3 files in 1 shot     |  |\n|  |                                   (Total: 5,445 input tokens)     |  |\n|  +-------------------------------------------------------------------+  |\n+-------------------------------------------------------------------------+\n```\n\n`RawTerminalOutput`): How 2 Test Runs Burned 92,385 Input Tokens\nTracking `execution-traces.json` and `usage.json` reveals how fast raw terminal output compounds in an agent's message array:\n\n`2,406 in / 10 out`):`{\"tool\": \"run_raw_pytest\"}`. The runner executes the 80-test suite and returns `60,749 characters``[PASS]` lines with `AUDIT_TRACE` dumps, and long multi-frame tracebacks for the 4 failures.`31,170 in / 484 out`):`+28,764 tokens`` 2,406` ➔ `31,170`). Distracted by the 60KB wall of text, the agent only patches `validator.py` (`stage_partial_fix`), triggering a second test run that appends another `56,544 characters`` 58,809 in / 1,574 out`):` 58,809 input tokens``92,385` cumulative input tokens).`TokenKillerFilter`): 1,506 Characters of Pure Signal\nWhen Mode B called `{\"tool\": \"run_filtered_tests\"}` on Turn 1, the harness preprocessor stripped all 76 passing tests, ANSI escape sequences, and `DEBUG` noise—returning a **`1,506-character` (`615-token`) JSON payload** containing only the 4 failing tests (` test`, `file`, `line`, `assertion`, `expected_vs_actual`):\n\n```\n{\n  \"summary\": {\"total\": 80, \"passed\": 76, \"failed\": 4, \"filtered_noise_chars_removed\": 59243},\n  \"failures\": [\n    {\n      \"test\": \"test_validator_contract\",\n      \"file\": \"validator.py\",\n      \"line\": 14,\n      \"assertion\": \"ALLOWED_CURRENCIES must include 'JPY'; reject empty list & bool; allow unit_cents==0 & chargeback_bps==1000; do not mutate input dict.\"\n    },\n    {\n      \"test\": \"test_settlement_ordering_and_reserve\",\n      \"file\": \"settlement.py\",\n      \"line\": 22,\n      \"assertion\": \"Preserve first-seen (merchant_id, currency) order (never sorted(order)); apply 10% reserve strictly when chargeback_bps > 300.\"\n    },\n    {\n      \"test\": \"test_settlement_discounts_and_rebate_cap\",\n      \"file\": \"settlement.py\",\n      \"line\": 39,\n      \"assertion\": \"Discount boundary is subtotal >= 15000 (GROWTH 10%, ENTERPRISE 20% else 10%); cap rebate_applied = min(fee_rebate, after_discount).\"\n    },\n    {\n      \"test\": \"test_statement_format_and_signed_zero\",\n      \"file\": \"statement.py\",\n      \"line\": 11,\n      \"assertion\": \"Use '#' delimiter ({merchant_id}#{currency}:) and preserve leading '-' on zero discount/rebate (TIER_DISCOUNT: -0, REBATE_APPLIED: -0).\"\n    }\n  ]\n}\n```\n\nBecause all four failure root causes were immediately visible without scrolling through 76 passing tests, Mode B fixed all three files in a single Turn 2 call (`3,030` input tokens)—eliminating the intermediate rework turn entirely.\n\n`labjudge16`)\nBoth modes passed all 20 hidden unit and E2E test cases verified by `labjudge16`:\n\n| # | Test Case ID | Invariant / Boundary Verified | Mode A ( `Raw CLI` ) | Mode B ( `Token Killer` ) | \n|---|---|---|---|---|\n| 1–6 | `validator_01` to`validator_06` | Trim/uppercase, non-mutation, `JPY` ,`bool` rejection,`0` cents &`1000` bps bounds | `6 / 6` PASS | **`6 / 6` PASS** | \n| 7–15 | `settlement_01` to`settlement_09` | First-seen order, conflict check, `300 vs 301` bps,`14999 vs 15000` tier, rebate cap | `9 / 9` PASS | **`9 / 9` PASS** | \n| 16–20 | `e2e_01` to`e2e_05` | `#` separator, signed zero`-0` ,`$0` auth statement, line order, error propagation | `5 / 5` PASS | **`5 / 5` PASS** | \n| **Total** | **20 Hidden Cases** | — | **`20 / 20` (100%)** | **`20 / 20` (100%)** | \n\nNever wire raw `subprocess.run([\"pytest\"])` stdout directly into your LLM agent's tool response. Instead:\n\n`pytest --tb=short -q --no-header --no-summary` (or `pytest-json-report`) inside your tool wrapper.`\"passed\": 500` as a single integer plus the exact file, line number, and assertion diff for the 2 failures.`rtk` (`Rust Token Killer`) or a 30-line Python wrapper around your test runner immediately cut agent token bills by `E20`):` MEMORY.md``E21`):", "url": "https://wpnews.pro/news/stop-dumping-raw-pytest-logs-into-your-coding-agent-how-a-structured-json-test-1", "canonical_source": "https://dev.to/tappers_/stop-dumping-raw-pytest-logs-into-your-coding-agent-how-a-structured-json-test-filter-cuts-input-451f", "published_at": "2026-11-01 01:16:54+00:00", "updated_at": "2026-11-01 01:21:46.857356+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["AI Labo", "Proxmox", "pytest", "SWE-agent", "gemini-3.8-flash", "rtk-ai/rtk"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/stop-dumping-raw-pytest-logs-into-your-coding-agent-how-a-structured-json-test-1", "markdown": "https://wpnews.pro/news/stop-dumping-raw-pytest-logs-into-your-coding-agent-how-a-structured-json-test-1.md", "text": "https://wpnews.pro/news/stop-dumping-raw-pytest-logs-into-your-coding-agent-how-a-structured-json-test-1.txt", "jsonld": "https://wpnews.pro/news/stop-dumping-raw-pytest-logs-into-your-coding-agent-how-a-structured-json-test-1.jsonld"}}