💡 AI Labo Benchmark Series — Experiment E21 (Ranked #2 out of 41 Patterns)
This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.
E21 (Backlog Candidate C017) e21-20261028-vm225-01
VM225 (ai-e21-225, 0 NICs, virtio-serial QGA relay, systemd UID separation)gemini-3.8-flash (thinkingLevel: LOW, temperature: 0.0)
When an AI coding agent edits a module that depends on several peripheral files, a common approach is to dump the entire source code of all imported dependencies into the prompt so the model "has full context."
In practice, feeding full implementation bodies of peripheral dependencies is actively counterproductive. Legacy helper logic, internal comments, and verbose function bodies bloat the context window and trigger Attention Dilution, causing the model to miss strict boundary contracts in the files it actually needs to edit.
To quantify this, we benchmarked a 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) inside isolated Proxmox VM VM225 across two harness modes:
FullDependencyContextRunner): 78,719 chars of tool output across 3 turns).TreeSitterSkeletonContextRunner):...) using an AST / Tree-sitter projection, injecting +673 tokens on Turn 1).
Passing only the Tree-sitter Interface Skeleton (2,316 input tokens) enabled a 1-turn 20 / 20 (100%) pass on the hidden test suite. In contrast, Mode A's full dependency dump consumed 45,270 input tokens across 3 turns and stalled at 17 / 20 (85.0%) due to attention dilution. Mode B cut cumulative input tokens by -94.9% (19.55x reduction), API cost by -82.6% ($0.0446 ➔ $0.0078), and execution time by -57.2% (2.34x faster).
| Metric | Mode A: Full Dependency Context ( FullDependencyContext ) |
Mode B: Tree-sitter Skeleton Context ( TreeSitterSkeleton ) |
Delta / Improvement |
|---|---|---|---|
| Hidden Grading Suite Score (20 Cases) | 17 / 20 (85.0% — missed 3 edge cases) |
20 / 20 (100% Perfect Pass) |
+15.0 pt (Zero defects) |
Contract Mismatch Traps Hit (gotcha_traps_hit) |
2 times (Turn 1, Turn 2) |
0 times (100% contract compliance) |
-100.0% (Zero rework) |
LLM Calls (model_calls) |
3 calls |
1 call (1-shot completion) |
-66.7% (3.00x faster convergence) |
Total File Edits (file_edits) |
5 edits |
3 edits (1 per module) |
-40.0% reduction |
Tool Output Volume (tool_output_chars) |
78,719 chars |
105 chars |
-99.9% (750x less noise) |
| Turn 1 Input / Output Tokens | 1,643 in /12 out |
2,316 in / 1,611 out |
Skeleton injection is only +673 in |
| Turn 2 Input / Output Tokens | 14,712 in /1,249 out |
— (Completed on Turn 1) | Avoided Turn 1 dependency dump bloat |
| Turn 3 Input / Output Tokens | 28,915 in /1,578 out |
— (Completed on Turn 1) | Avoided Turn 2 cumulative log bloat |
Cumulative Input Tokens (promptTokenCount) |
45,270 tokens |
2,316 tokens |
-94.9% (19.55x token efficiency) |
Cumulative Output Tokens (candidatesTokenCount) |
2,839 tokens |
1,611 tokens |
-43.3% reduction |
Elapsed Execution Time (elapsed_seconds) |
29.99 sec |
12.84 sec |
-57.2% (2.34x faster) |
Estimated API Cost (estimated_micro_usd) |
$0.0446 (44,600 micro-USD ) |
$0.0078 (7,779 micro-USD) |
-82.6% (5.73x cheaper) |
+-------------------------------------------------------------------------+
| Proxmox Isolated VM225 (0 NICs, virtio-serial QGA, systemd UID split) |
| Target Task: 3-Module Settlement Pipeline Refactoring |
| (validator.py / settlement.py / statement.py) |
| |
| +-------------------------------------------------------------------+ |
| | Mode A: FullDependencyContextRunner (Unfiltered Full Source Dump) | |
| | - Turn 1: load_full_dependency_dump (1,643 in / 39,373 chars) | |
| | - Turn 2: stage_dependency_patch (14,712 in / 39,027 chars) | |
| | - Turn 3: submit_skeleton_guided_bundle (28,915 in --> 17 / 20) | |
| +-------------------------------------------------------------------+ |
| |
| +-------------------------------------------------------------------+ |
| | Mode B: TreeSitterSkeletonContextRunner (Signature & Contract) | |
| | - Turn 1: Reads AST Interface Skeleton (+673 tok) & submits all | |
| | 3 modules (2,316 in / 1,611 out --> 20 / 20 on Turn 1) | |
| +-------------------------------------------------------------------+ |
| | |
| v |
| +-------------------------------------------------------------------+ |
| | Independent Grader (labjudge21): 20 Hidden Unit & E2E Test Cases | |
| +-------------------------------------------------------------------+ |
+-------------------------------------------------------------------------+
input-03.json)
Instead of dumping internal helper functions, Mode B's harness projected the target modules and their contracts into a compact Interface Skeleton (2,316 total input tokens on Turn 1) with implementation bodies folded into ...:
ALLOWED_CURRENCIES: tuple[str, ...] = ("USD", "EUR", "JPY")
def validate_txns(raw_txns: list[dict]) -> list[dict]:
"""Contract:
1. Reject empty list or non-list with ValueError.
2. Strip & uppercase `merchant_id` (non-empty str) and `currency` (must be in ALLOWED_CURRENCIES: USD, EUR, JPY).
3. Reject `isinstance(v, bool)` before `isinstance(v, int)` on `tx_count`, `unit_cents`, `chargeback_bps`.
4. Bounds: `tx_count > 0`, `unit_cents >= 0` (zero allowed!), `0 <= chargeback_bps <= 1000` (1000 inclusive!).
5. Return a brand-new 5-key dict list without mutating caller dicts.
"""
...
ALLOWED_TIERS: tuple[str, ...] = ("STANDARD", "GROWTH", "ENTERPRISE")
def settle_batch(txns: list[dict], tier: str = "STANDARD", fee_rebate: int = 0) -> dict:
"""Contract:
1. Preserve first-seen `(merchant_id, currency)` order (`for k in order:`, NEVER `sorted(order)`).
2. Chargeback reserve: 10% (`(gross * 10) // 100`) STRICTLY when `chargeback_bps > 300` (at 300 bps, reserve == 0).
3. Tier discount threshold `subtotal >= 15000` is INCLUSIVE:
- `GROWTH`: 10% if `subtotal >= 15000` else 0.
- `ENTERPRISE`: 20% if `subtotal >= 15000` else 10%.
4. Rebate cap: `rebate_applied = min(fee_rebate, after_discount)`.
"""
...
def render_statement(batch: dict) -> str:
"""Contract:
1. Line format: `{merchant_id}#{currency}: tx={tx_count} unit={unit_cents} bps={chargeback_bps} = {line_total}` (delimiter is `#`, NOT `:`).
2. Discount & rebate lines MUST always carry a leading `-` sign even when 0 (`TIER_DISCOUNT: -0`, `REBATE_APPLIED: -0`).
"""
...
In Mode A (FullDependencyContextRunner), calling load_full_dependency_dump on Turn 1 and stage_dependency_patch on Turn 2 flooded the context with 39,373 chars and 39,027 chars of internal helper bodies, bitwise checksum loops, and deprecated v1 comments:
=== FULL DEPENDENCY IMPLEMENTATION DUMP & INTERFACE MISMATCH LOG (STAGE #1) ===
Status: MODE_A_TRAP_HIT (4 contract invariants unresolved in FullDependencyContextRunner)
def _internal_helper_pipeline_stage_01(payload_buffer, flags=17):
"""Internal helper implementation body #01 (non-public implementation details)."""
acc_state = 1024
for step_i in range(16):
acc_state = ((acc_state ^ (step_i * 31)) + 97) & 0xFFFFFFFF
return {'stage': 1, 'checksum': hex(acc_state), 'legacy_delimiter': ':'}
... (37,500+ chars of internal helper implementation bodies) ...
usage.json)
| Seq | Mode | Node ID | Input Tokens | Output Tokens | API Cost ( micro-USD ) |
|---|---|---|---|---|---|
0 |
Mode A ( full_dep_context ) |
full_dep_context_turn_1 |
1,643 |
12 |
1,278 ($0.0013 ) |
1 |
Mode A ( full_dep_context ) |
full_dep_context_turn_2 |
14,712 |
1,249 |
15,718 ($0.0157 ) |
2 |
Mode A ( full_dep_context ) |
full_dep_context_turn_3 |
28,915 |
1,578 |
27,604 ($0.0276 ) |
3 |
Mode B (treesitter_skeleton) |
treesitter_skeleton_turn_1 |
2,316 |
1,611 |
7,779 ($0.0078) |
| Total | Mode A (3 Turns) | — | 45,270 |
2,839 |
44,600 ($0.0446) |
| Total | Mode B (1 Turn) | — | 2,316 |
1,611 |
7,779 ($0.0078) |
Inspecting the final submitted bundles (full-dep-bundle.json vs. treesitter-skeleton-bundle.json) shows how 28,915 tokens of noisy dependency implementations diluted Mode A's attention on Turn 3:
if cnt <= 0 or cents <= 0 or not (0 <= bps <= 1000):
raise ValueError("numeric bounds violated")
out.append({
**item,
"merchant_id": mid.strip().upper(),
"currency": cur.strip().upper(),
})
if cnt <= 0 or cents < 0 or not (0 <= bps <= 1000):
raise ValueError("numeric bounds violated")
out.append({
"merchant_id": mid.strip().upper(),
"currency": cur.strip().upper(),
"tx_count": cnt,
"unit_cents": cents,
"chargeback_bps": bps,
})
Buried under 78KB of irrelevant helper bodies, Mode A missed two critical boundary contracts in validator.py:
validator_01_normalize_and_non_mutation FAIL):{**item, ...} copied unvalidated extra keys (e.g., "extra": 99) from the input dictionary into the normalized output instead of returning a strict 5-key dictionary.validator_05 & e2e_03 FAIL):cents <= 0 instead of cents < 0, rejecting valid $0 card authorization probes (unit_cents == 0).
Mode B, operating on a crystal-clear 2,316-token AST interface skeleton, passed all 20 / 20 test cases on its very first turn.
When your agent needs to call or integrate with neighboring modules that it is not actively modifying, never dump their full source code into the prompt. Instead, use Python's ast module (or tree-sitter for multi-language repos) to fold function bodies into ...:
import ast
def extract_interface_skeleton(source_code: str) -> str:
"""Fold function and method bodies into '...' while preserving signatures, types, and docstrings."""
tree = ast.parse(source_code)
for node in ast.walk(tree):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
has_doc = (
node.body
and isinstance(node.body[0], ast.Expr)
and isinstance(node.body[0].value, ast.Constant)
and isinstance(node.body[0].value.value, str)
)
ellipsis_stmt = ast.Expr(value=ast.Constant(value=...))
node.body = [node.body[0], ellipsis_stmt] if has_doc else [ellipsis_stmt]
return ast.unparse(tree)
By exposing interfaces and contracts while hiding internal implementation noise, you cut input tokens by over 90% and eliminate Lost-in-the-Middle bugs.
E20): MEMORY.md