I Stopped Trusting My AI Coding Agent. So I Touched some grass & Built something to Watch It

A developer built a policy-based monitoring system for AI coding agents after a Claude Code session unexpectedly modified a config file without visibility. The system uses taint tracking to detect sensitive data crossing trust boundaries and integrates with OpenTelemetry and SigNoz for trace-based observability. The developer plans to extend it to real Claude Code sessions via the PreToolUse hook.

I use Claude Code daily. Mostly it's a cool tool. A few weeks back while refactoring a large & messy bug-bounty codebase, it also touched a config file I never asked it to touch. I only caught it because I happened to see it while pushing the code to github, I saw a file appear in commit history which I never touch. That's what pissed me, not the mistake, the lack of visibility . In a codebase that size, I can't manually catch every unintended write, every file it peeked into it didn't need. So instead of asking "is this agent helpful," I started asking a security question: what is it actually doing, and who's watching? .env , an admin module do I get alerted I built this on a toy agent Like my ex used me for fun first a minimal loop with four tools read file , write file , run shell , call api , each one gated through a policy layer before it's allowed to execute. Two jobs: tag sensitive data the moment the agent touches it, and block/pause anything that tries to carry tagged data across a trust boundary. lets see some codes now php policy.py - the actual detection logic, unedited SECRET RE = re.compile r" sk- a-zA-Z0-9 {16,}|AKIA 0-9A-Z {16}|AIza 0-9A-Za-z\- {35}|" r"ghp a-zA-Z0-9 {20,}|Bearer\s+ a-zA-Z0-9\- \. {20,}|" r" a-fA-F0-9 {32,}|-----BEGIN A-Z +PRIVATE KEY----- " def evaluate call tool name, target, params, inherited tags : if tool name == "run shell" and DANGEROUS SHELL RE.search target : return PolicyResult Decision.BLOCK, 100, "dangerous shell pattern" if Tag.SECRET in inherited tags and tool name in "call api", "write file" and is external target : return PolicyResult Decision.BLOCK, 95, "secret data exfil attempt" if Tag.PII in inherited tags and tool name in "call api", "write file" and is external target : return PolicyResult Decision.PENDING CONFIRM, 70, "pii crossing trust boundary" return PolicyResult Decision.ALLOW, 0, The taint tags carry forward across steps if step 1 reads a file with PII in it, step 3 still knows that, even though it's a completely different tool call. That's the part that matters: not "was this one action dangerous," but "did sensitive data quietly travel somewhere it shouldn't have, across a chain of individually-innocent steps." Here's it actually running, unedited output from my terminal: classify content "Contact me at john@example.com for details" 'pii' evaluate call "run shell", "rm -rf /", {}, set block 'dangerous shell pattern' evaluate call "call api", "https://evil-domain.com/exfil", {}, {Tag.SECRET} block 'secret data exfil attempt' Every decision above gets wrapped in an OpenTelemetry span and shipped to SigNoz, self-hosted: instrumentation.py with tracer.start as current span f"tool.{tool name}" as span: span.set attribute "tool.name", tool name span.set attribute "tool.target", target span.set attribute "data.tags", ",".join taint ctx.snapshot span.set attribute "risk.score", policy result.risk score span.set attribute "policy.decision", policy result.decision.value A log line tells you something happened. A trace tells you the shape of what happened — the whole session as a waterfall, in order, so I can open one session and see exactly what it touched and why. That's the actual reason I picked SigNoz over just writing to a log file: A toy agent proves the mechanism. It doesn't prove anything about the tool I actually use daily. So the next step is pointing this at real Claude Code sessions, using its PreToolUse hook which can fire before any tool executes and deny it outright, including over plain HTTP: // settings.json draft { "hooks": { "PreToolUse": { "matcher": "Read|Edit|Write", "hooks": { "type": "http", "url": "http://localhost:8080/hooks/pre-tool-use", "timeout": 10 } } } } My server on the other end reuses evaluate call unchanged, just fed from Claude Code's JSON payload instead of my toy agent's. The policy for the repo I'm working in right now is blunt: don't read .env , don't touch admin/ , don't write to config without asking and tell me the instant it tries. Toy-agent version: built, tested, working the output above is real. Claude Code integration: in progress. I'm publishing this mid-build because the honest version is more useful than the polished one. If you've ever wondered what your coding agent actually touched last session you probably don't know either. That's the gap this closes. Touch some grass before your agent touches something that It should not. Building something similar, or think my regex-based PII detector is going to embarrass me in front of judges? I'd like to hear about it.