We Traced Prime Agent 20 Times. Your Working Directory Is Invisible Input.

A solo developer built AgentPulse, a metadata-only observability and evaluation layer for production AI agents that records execution shape—span counts, timing, token usage and approved labels—without capturing prompts, outputs, tool inputs or secrets. In a 20-run validation on Prime Agent using gpt-5.4, the same task in a cluttered working directory produced a median of 23 spans versus 6 in an empty folder, a 3.8x increase attributable to environment alone. The evidence adapter returned a verdict of "noise" (confidence 0.6) rather than "regression," since claiming a regression requires 20 traces per set and only 10 were available.

This is a submission for the MLH x DEV Writing Challenge https://dev.to/mlh-hackathon I built AgentPulse , a metadata-only observability and evaluation layer for production AI agents. It never captures prompts, outputs, tool inputs, or secrets — only the shape of execution span counts, timing, token usage, approved technical labels . This week I validated the eval layer on Prime Agent with 20 runs: same task, same model gpt-5.4 , only the working directory changed. Baseline empty folder : median 6 spans. Candidate cluttered folder : median 23 spans. 3.8x increase from folder contents alone. The evidence adapter verdict: noise confidence 0.6 — not regression , because it requires 20 traces/set to claim regression. Correct conservative call on n=10. What I learned: 1 Trace shape is a behavioral fingerprint — no prompts needed. 2 Environment is input — working directory can 3.8x your costs. 3 Conservative evals build trust. Session Spans LLM Tool Time Set -------- ----- --- ---- ----- ---- 01a1a806 8 4 3 14:26 BASE 01a1a80a 8 4 3 14:30 BASE 01a1a80c 6 3 2 14:32 BASE 01a1a80e 4 2 1 14:34 BASE 01a1a80f 8 4 3 14:36 BASE 01a1a811 6 3 2 14:38 BASE 01a1a813 6 3 2 14:40 BASE 01a1a815 8 4 3 14:42 BASE 01a1a817 4 2 1 14:44 BASE 01a1a819 2 1 0 14:46 BASE 01a1a81a 12 6 5 14:49 CAND 01a1a81d 30 15 14 14:52 CAND 01a1a820 12 6 5 14:55 CAND 01a1a823 24 12 11 14:58 CAND 01a1a826 22 11 10 15:01 CAND 01a1a828 30 15 14 15:05 CAND 01a1a82b 22 11 10 15:08 CAND 01a1a82e 32 16 15 15:11 CAND 01a1a831 36 18 17 15:14 CAND 01a1a834 2 1 0 15:16 CAND Adapter output: verdict noise , median shift 17.0, IQR ratio 4.5. Didn't use MLH partner tech — built on open-source Python, TypeScript, Docker, ClickHouse, Postgres with OpenAI API for the experiment. The value is in the methodology. Didn't attend a hackathon — this is a solo build. But the most memorable moment was seeing 30+ span runs come back, and watching the adapter say noise instead of regression when I wanted the stronger verdict. That's when I knew the eval layer was working — it protects you from confirmation bias.