This is a submission for the MLH x DEV Writing Challenge
I built AgentPulse, a metadata-only observability and evaluation layer for production AI agents. It never captures prompts, outputs, tool inputs, or secrets — only the shape of execution (span counts, timing, token usage, approved technical labels).
This week I validated the eval layer on Prime Agent with 20 runs: same task, same model (gpt-5.4), only the working directory changed. Baseline (empty folder): median 6 spans. Candidate (cluttered folder): median 23 spans. 3.8x increase from folder contents alone.
The evidence adapter verdict: noise (confidence 0.6) — not regression, because it requires 20 traces/set to claim regression. Correct conservative call on n=10.
What I learned: (1) Trace shape is a behavioral fingerprint — no prompts needed. (2) Environment is input — working directory can 3.8x your costs. (3) Conservative evals build trust.
Session Spans LLM Tool Time Set
-------- ----- --- ---- ----- ----
01a1a806 8 4 3 14:26 BASE
01a1a80a 8 4 3 14:30 BASE
01a1a80c 6 3 2 14:32 BASE
01a1a80e 4 2 1 14:34 BASE
01a1a80f 8 4 3 14:36 BASE
01a1a811 6 3 2 14:38 BASE
01a1a813 6 3 2 14:40 BASE
01a1a815 8 4 3 14:42 BASE
01a1a817 4 2 1 14:44 BASE
01a1a819 2 1 0 14:46 BASE
01a1a81a 12 6 5 14:49 CAND
01a1a81d 30 15 14 14:52 CAND
01a1a820 12 6 5 14:55 CAND
01a1a823 24 12 11 14:58 CAND
01a1a826 22 11 10 15:01 CAND
01a1a828 30 15 14 15:05 CAND
01a1a82b 22 11 10 15:08 CAND
01a1a82e 32 16 15 15:11 CAND
01a1a831 36 18 17 15:14 CAND
01a1a834 2 1 0 15:16 CAND
Adapter output: verdict noise, median shift 17.0, IQR ratio 4.5.
Didn't use MLH partner tech — built on open-source (Python, TypeScript, Docker, ClickHouse, Postgres) with OpenAI API for the experiment. The value is in the methodology.
Didn't attend a hackathon — this is a solo build. But the most memorable moment was seeing 30+ span runs come back, and watching the adapter say noise instead of regression when I wanted the stronger verdict. That's when I knew the eval layer was working — it protects you from confirmation bias.__