{"slug": "we-traced-prime-agent-20-times-your-working-directory-is-invisible-input", "title": "We Traced Prime Agent 20 Times. Your Working Directory Is Invisible Input.", "summary": "A solo developer built AgentPulse, a metadata-only observability and evaluation layer for production AI agents that records execution shape—span counts, timing, token usage and approved labels—without capturing prompts, outputs, tool inputs or secrets. In a 20-run validation on Prime Agent using gpt-5.4, the same task in a cluttered working directory produced a median of 23 spans versus 6 in an empty folder, a 3.8x increase attributable to environment alone. The evidence adapter returned a verdict of \"noise\" (confidence 0.6) rather than \"regression,\" since claiming a regression requires 20 traces per set and only 10 were available.", "body_md": "*This is a submission for the [MLH x DEV Writing Challenge](https://dev.to/mlh-hackathon)*\n\nI built **AgentPulse**, a metadata-only observability and evaluation layer for production AI agents. It never captures prompts, outputs, tool inputs, or secrets — only the shape of execution (span counts, timing, token usage, approved technical labels).\n\nThis week I validated the eval layer on Prime Agent with 20 runs: same task, same model (gpt-5.4), only the working directory changed. Baseline (empty folder): median 6 spans. Candidate (cluttered folder): median 23 spans. **3.8x increase** from folder contents alone.\n\nThe evidence adapter verdict: `noise` (confidence 0.6) — not `regression`, because it requires 20 traces/set to claim regression. Correct conservative call on n=10.\n\nWhat I learned: (1) Trace shape is a behavioral fingerprint — no prompts needed. (2) Environment is input — working directory can 3.8x your costs. (3) Conservative evals build trust.\n\n```\nSession    Spans  LLM  Tool  Time   Set\n--------   -----  ---  ----  -----  ----\n01a1a806   8      4    3     14:26  BASE\n01a1a80a   8      4    3     14:30  BASE\n01a1a80c   6      3    2     14:32  BASE\n01a1a80e   4      2    1     14:34  BASE\n01a1a80f   8      4    3     14:36  BASE\n01a1a811   6      3    2     14:38  BASE\n01a1a813   6      3    2     14:40  BASE\n01a1a815   8      4    3     14:42  BASE\n01a1a817   4      2    1     14:44  BASE\n01a1a819   2      1    0     14:46  BASE\n01a1a81a   12     6    5     14:49  CAND\n01a1a81d   30     15   14    14:52  CAND\n01a1a820   12     6    5     14:55  CAND\n01a1a823   24     12   11    14:58  CAND\n01a1a826   22     11   10    15:01  CAND\n01a1a828   30     15   14    15:05  CAND\n01a1a82b   22     11   10    15:08  CAND\n01a1a82e   32     16   15    15:11  CAND\n01a1a831   36     18   17    15:14  CAND\n01a1a834   2      1    0     15:16  CAND\n```\n\nAdapter output: verdict `noise`, median shift 17.0, IQR ratio 4.5.\n\nDidn't use MLH partner tech — built on open-source (Python, TypeScript, Docker, ClickHouse, Postgres) with OpenAI API for the experiment. The value is in the methodology.\n\nDidn't attend a hackathon — this is a solo build. But the most memorable moment was seeing 30+ span runs come back, and watching the adapter say `noise` instead of `regression` when I wanted the stronger verdict. That's when I knew the eval layer was working — it protects you from confirmation bias.__", "url": "https://wpnews.pro/news/we-traced-prime-agent-20-times-your-working-directory-is-invisible-input", "canonical_source": "https://dev.to/vishwajitsinh_mahida_2547/we-traced-prime-agent-20-times-your-working-directory-is-invisible-input-35f3", "published_at": "2026-11-04 20:37:09+00:00", "updated_at": "2026-11-04 20:53:23.352306+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "ai-tools", "ai-safety"], "entities": ["AgentPulse", "Prime Agent", "gpt-5.4", "OpenAI", "ClickHouse", "Postgres", "Docker"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/we-traced-prime-agent-20-times-your-working-directory-is-invisible-input", "markdown": "https://wpnews.pro/news/we-traced-prime-agent-20-times-your-working-directory-is-invisible-input.md", "text": "https://wpnews.pro/news/we-traced-prime-agent-20-times-your-working-directory-is-invisible-input.txt", "jsonld": "https://wpnews.pro/news/we-traced-prime-agent-20-times-your-working-directory-is-invisible-input.jsonld"}}