# We Traced Prime Agent 20 Times. Your Working Directory Is Invisible Input. > Source: > Published: 2026-11-04 20:37:09+00:00 *This is a submission for the [MLH x DEV Writing Challenge](https://dev.to/mlh-hackathon)* I built **AgentPulse**, a metadata-only observability and evaluation layer for production AI agents. It never captures prompts, outputs, tool inputs, or secrets — only the shape of execution (span counts, timing, token usage, approved technical labels). This week I validated the eval layer on Prime Agent with 20 runs: same task, same model (gpt-5.4), only the working directory changed. Baseline (empty folder): median 6 spans. Candidate (cluttered folder): median 23 spans. **3.8x increase** from folder contents alone. The evidence adapter verdict: `noise` (confidence 0.6) — not `regression`, because it requires 20 traces/set to claim regression. Correct conservative call on n=10. What I learned: (1) Trace shape is a behavioral fingerprint — no prompts needed. (2) Environment is input — working directory can 3.8x your costs. (3) Conservative evals build trust. ``` Session Spans LLM Tool Time Set -------- ----- --- ---- ----- ---- 01a1a806 8 4 3 14:26 BASE 01a1a80a 8 4 3 14:30 BASE 01a1a80c 6 3 2 14:32 BASE 01a1a80e 4 2 1 14:34 BASE 01a1a80f 8 4 3 14:36 BASE 01a1a811 6 3 2 14:38 BASE 01a1a813 6 3 2 14:40 BASE 01a1a815 8 4 3 14:42 BASE 01a1a817 4 2 1 14:44 BASE 01a1a819 2 1 0 14:46 BASE 01a1a81a 12 6 5 14:49 CAND 01a1a81d 30 15 14 14:52 CAND 01a1a820 12 6 5 14:55 CAND 01a1a823 24 12 11 14:58 CAND 01a1a826 22 11 10 15:01 CAND 01a1a828 30 15 14 15:05 CAND 01a1a82b 22 11 10 15:08 CAND 01a1a82e 32 16 15 15:11 CAND 01a1a831 36 18 17 15:14 CAND 01a1a834 2 1 0 15:16 CAND ``` Adapter output: verdict `noise`, median shift 17.0, IQR ratio 4.5. Didn't use MLH partner tech — built on open-source (Python, TypeScript, Docker, ClickHouse, Postgres) with OpenAI API for the experiment. The value is in the methodology. Didn't attend a hackathon — this is a solo build. But the most memorable moment was seeing 30+ span runs come back, and watching the adapter say `noise` instead of `regression` when I wanted the stronger verdict. That's when I knew the eval layer was working — it protects you from confirmation bias.__