I Shipped an Agent Gatekeeper (v0.1). 14 Developers Showed Me What I Missed. Here's v0.2 — a Control Plane.

Developer Deghosal shipped v0.2.0 of agent-tooltrust, an open-source control plane for AI agents, three days after field-testing v0.1.0 with 83 real agents and incorporating feedback from 14 developers. The update adds argument-level policy, resource/environment scoping, conditional allows with obligations, three-tier deny-reason exposure, and premise/staleness validation, transforming the tool from a gatekeeper into a control plane.

Previously: I Stopped Trusting AI Agents With Tools. So I Built a Gatekeeper. — the v0.1.0 launch story Aug 13 . Code: github.com/deghosal-2026/agent-tooltrust · pip install agent-tooltrust v0.2.0 Release: v0.2.0 on GitHub · PyPI Three days ago I shipped a gatekeeper for AI agents. The idea was simple: before an agent's tool call executes, a deterministic engine scores it across five risk dimensions and returns one of four decisions — allow, audit, escalate, or deny. The engine sits outside the model, not inside the prompt. No amount of prompt engineering overrides a deny. I field-tested it against 83 real agents across 10 frameworks. Published to PyPI. Made the repo public. Wrote a dev.to article https://dev.to/debashish ghosal/i-stopped-trusting-ai-agents-with-tools-so-i-built-a-gatekeeper-26fb about it. Thought I was done. I wasn't even close. The article got attention, and the comments weren't "nice project." They were sharp, specific, and uncomfortable. People who actually deploy agents read what I'd built and said: this is good, but you're missing the thing that actually breaks in production. That's the thing about shipping. You think you've built the product. Then real users tell you what the product actually is. Every comment was a gap I didn't see — not because I didn't plan well, but because you can't see your own blind spots. You need someone else to point at them. 14 comments turned into 14 GitHub issues. Every single one shipped in v0.2.0, three days later. The full list is here https://github.com/deghosal-2026/agent-tooltrust/issues?q=is%3Aissue+milestone%3A%22v0.2.0%22+is%3Aclosed . This is what happened in those three days — what the feedback became, what the field test taught me about local vs cloud LLMs, and how a gatekeeper became a control plane. I expected "cool project" and maybe a star. Instead I got three categories of feedback that completely reshaped what v0.2.0 became. v0.1.0 validated tool selection — is this agent allowed to call this tool? Several developers pointed out that's barely the surface. One comment said: you validate that DELETE is allowed, but what about the arguments? A DELETE with no WHERE clause is dangerous even if DELETE itself is permitted. That became argument-level policy https://github.com/deghosal-2026/agent-tooltrust/issues/142 — per-tool argument schemas with required fields, forbid-lists, bounds, and environment allowlists, all evaluated before the engine decides. Another said: an agent scoped to staging shouldn't be able to touch production. That became resource/environment scoping https://github.com/deghosal-2026/agent-tooltrust/issues/145 — default-deny on out-of-scope resolution, session-scoped environments. Someone else asked about conditional allows — what if a call should be allowed but only with a sign-off or a mandatory notification? That became allow with obligation https://github.com/deghosal-2026/agent-tooltrust/issues/147 — a fifth decision state where the gatekeeper enforces side-effects first-use sign-off, auto-notify, signed audit entry that fire even if the agent doesn't cooperate. Then there was the deny-explanation problem. If you deny and explain too much, the agent can reverse-engineer your rule set. If you don't explain enough, it can't replan. That became three-tier deny-reason exposure https://github.com/deghosal-2026/agent-tooltrust/issues/160 — none , reason-only , detail — with reason-only as the default so agents get enough to replan without seeing rule IDs or thresholds. And finally: schema-valid and permission-valid calls can still be semantically wrong. Reading a moved file path. Calling the wrong tool with plausible arguments. That became premise/staleness validation https://github.com/deghosal-2026/agent-tooltrust/issues/159 . v0.1.0 trusted the agent's self-report. Multiple developers asked the same question in different ways: why? A burst of denies isn't just noise — it's a probe. Someone reverse-engineering your policy. That became deny-storm detection https://github.com/deghosal-2026/agent-tooltrust/issues/143 — a session-level analyzer tracking deny rate, consecutive denies, and tool-set entropy, with throttle/lock/pause on threshold. URL fetches can exfiltrate. An agent fetching 169.254.169.254 is reading cloud metadata, not a website. That became the URL fetch category guard https://github.com/deghosal-2026/agent-tooltrust/issues/146 — robots.txt enforcement, PII stripping, and SSRF redirect re-resolution against internal-address blocklists. The agent says "done." How do you know it actually did the work? That became the external verification sink https://github.com/deghosal-2026/agent-tooltrust/issues/144 — read-only hooks to agent-unwritable systems API counters, VCS state, billing snapshots that diff against the agent's self-report. This one stung. v0.1.0 logged everything — every decision, every argument, every explanation. A developer pointed out: if an agent reads a secret in audit mode, that secret is now in your audit store. The gatekeeper that protects tools is persisting their contents. The audit trail became a secondary breach vector. That became audit argument redaction https://github.com/deghosal-2026/agent-tooltrust/issues/158 — a default deny-list token , password , apiKey , authorization , secret , … that redacts sensitive args recursively through nested dicts and lists, replaces them with REDACTED , sets a redacted: true flag, and supports per-policy overrides. Then there was the credential problem. A stale or out-of-scope credential at call time looks identical to "the LLM didn't call the tool" in the audit trail. You can't tell the difference between a policy failure and a credential failure. That became stale-credential classification https://github.com/deghosal-2026/agent-tooltrust/issues/161 — a credential status tag that distinguishes engine-allow-but-credential-rejected from not-available . Nobody was looking across sessions. A denied call last week that's allowed this week is drift — but v0.1.0 had no way to see it. That became session-to-session analytics https://github.com/deghosal-2026/agent-tooltrust/issues/148 — recurring benign denials, deny→allow transitions, dead and over-hit rules, all queryable via tooltrust analytics sessions and the /api/analytics/sessions endpoint. And the thresholds themselves were static. How do you know they're right? That became score calibration https://github.com/deghosal-2026/agent-tooltrust/issues/162 — counterfactual threshold logging what score would have flipped the decision , false-allow and false-escalate rates broken down by tool, environment, and data class, with tooltrust calibrate report and the /api/analytics/calibration endpoint. Two more comments pushed the test methodology itself. Add adversarial parameter payloads 163 https://github.com/deghosal-2026/agent-tooltrust/issues/163 . Test scenarios the gate cannot pass by construction 164 https://github.com/deghosal-2026/agent-tooltrust/issues/164 . Both shipped. 14 comments. 14 issues. 14 features. All from one dev.to article https://dev.to/debashish ghosal/i-stopped-trusting-ai-agents-with-tools-so-i-built-a-gatekeeper-26fb . Here's the thing that surprised me: the engine didn't change at all. Engine.evaluate returns the same decisions for the same inputs. The 2,490 deterministic tests from v0.1.0 still pass at 100%. The field test proved no regression. What changed is everything around the engine. Before it: argument validation, scope enforcement, premise checks. After it: redaction, calibration, stale-credential tagging, tamper-evident chaining. Around it: session analytics, HTTP PDP, policy packs, OPAL sync, and a 5-tab operator dashboard. The gatekeeper became a control plane. Not because I planned that — because 14 developers told me what was missing, and filling those gaps turned a per-call interceptor into a system you can observe, tune, and serve to a fleet. tooltrust audit session --replay <id reconstruct cumulative risk from audit tooltrust audit verify verify tamper-evident hash chain tooltrust analytics sessions deny→allow transitions, dead rules tooltrust calibrate report counterfactual thresholds, false rates tooltrust pack list community policy pack catalog tooltrust policy rollback --version <v roll back a policy version tooltrust baseline check hardened 15/15 security baseline checks HTTP POST /authorize returns Decision JSON — Go, JS, Java can now query the PDP without Python. An MCP-Data connector authorizes per data source. OPAL distributes policy updates across fleet instances with Engine.reload policy and rollback. Five seed policy packs ship in the catalog. The operator console is where the "control plane" framing becomes visible. You don't just see decisions — you review escalations, replay sessions, calibrate thresholds, and check fleet posture: Dashboard — fleet overview with decision counts, deny rate, active sessions: Escalations — pending human approvals, approve/deny from the UI: Audit — searchable decision log with full context per entry: Sessions — replay a session's cumulative risk at each call: Analytics — deny patterns, deny→allow transitions, calibration rates: Baselines — Hardened 15/15, OWASP 10/10, OpenSSF status: | Baseline | v0.1.0 | v0.2.0 | |---|---|---| | OWASP Agentic AI Top 10 | 5/10 | 10/10 | | ToolTrust Security Baseline | Essential | Hardened 15/15 | | OpenSSF | Silver | Path to Gold 12/14 | v0.1.0 ran the field test on a local 4B model Qwen3.5-4B-4bit via OMLX on Apple Silicon . It was slow — 30-80 seconds per agent call — and the 7 tier-1 failures were blamed on the weak local model. v0.2.0 re-ran the same field test on four models: gpt-oss-20b $0.03/M , deepseek-v4-flash $0.07/M , glm-5 similar price , and the same local Qwen 4B. The goal was to see if a better model would fix the tier-1 failures. It didn't. And that's when I realized something I'd gotten wrong in v0.1.0. I blamed the local 4B model for the 7 tier-1 failures. "The model is too weak," I wrote. "Use a bigger model." But when I ran the same tests against gpt-oss-20b, deepseek-v4-flash, and glm-5 — all smarter, all cloud, all faster — the same 7 failures showed up. Not the same scenarios failing on different models. The same pattern : the LLM picks the wrong tool when 5 tools have near-identical names. When an agent has exactly one scenario tool, every model — cheap cloud, expensive cloud, free local — reliably calls it. The guard fires, the decision is recorded, the row matches the golden expectation. 83/83 on Plan A, across all 4 models, with zero exceptions. This confirmed what I suspected but couldn't prove in v0.1.0: the engine is correct. The adapter wiring is correct. The failures are not about the engine or the adapters. The 7 tier-1 failures from v0.1.0 weren't caused by the weak local model. They're caused by the pattern. When an agent has 5 tools with near-identical names scn decision-allow-01 , scn decision-audit-01 , scn decision-escalate-01 , scn decision-deny-01 , scn adversarial-injection-01 , every model picks the wrong one sometimes: | Model | Tier-1 5 tools | Single-tool | |---|---|---| | gpt-oss-20b | 2/5 | 100% | | deepseek-v4-flash | 2/5 | 100% | | glm-5 | 4/5 best | 100% | | Qwen3.5-4B-4bit local | 2/5 | 100% | I tried strengthening the prompt: "call ONLY the tool, do not call any other tool." It made things worse — crew-01 went from 4/5 to 2/5. Tool selection for near-identical names isn't prompt-steerable. It's a retrieval/attention limitation in the models themselves. This is the kind of finding you only get by running the same test against multiple models. v0.1.0 had one model and one explanation "the 4B is weak" . v0.2.0 has four models and a different conclusion: no current model at this price point can reliably select from 5 similarly-named tools. The fix isn't a better model or a better prompt — it's a different test design. | Local Qwen 4B | Cloud glm-5 etc. | | |---|---|---| | Cost | Free | ~$0.30-$0.70/sweep | | Speed | 5-55s/agent | 3-12s/agent | | Single-tool | 100% | 100% | | Tier-1 | 2/5 | 2/5 - 4/5 | | Best for | Smoke tests, one agent | Full sweeps, retries | Cloud specifically glm-5 fixed 6 individual agents that the local model couldn't — but no model fixed tier-1. The practical takeaway: use cloud for full sweeps, local for quick smoke checks, and don't waste time trying to fix tier-1 by swapping models. This one cost me an hour. gpt-oss-20b sometimes returns an empty assistant message — no content, no tool call. The field test harness reads that as not-available the LLM didn't call the tool . But the model wasn't refusing — it was thinking . It's a reasoning model, and with a small max tokens budget, it spends the entire allocation on internal reasoning and returns nothing visible. Send it max tokens=200 and the content appears. If your agent harness treats empty responses as failures, you'll debug your integration when the actual problem is token budget allocation. Worth knowing before you spend an hour on it. The deterministic matrix proves the engine is correct. It can't prove the adapters work in real agent loops. The live field test caught three real deployment-breaking bugs: smolagents LiteLLM hang — the openai/ model prefix stalls silently on OpenRouter. No timeout, no error, just a hang. The fix was using the openrouter/ prefix so litellm routes through its native OpenRouter provider instead of the OpenAI provider. Scenario tool entry serialization — smolagents builds tool schemas from function signatures. The scenario tool had entry: dict = entry as a parameter, and smolagents serialized the bound dict containing a callable to {} . When invoked, entry "fn" raised KeyError . The fix was a closure factory that captures the callable without exposing it in the signature. Interactive deny retry loop — when the guard denies a call, it raises ToolTrustDecisionError . The smolagents agent loop catches that as a tool error and retries — up to max steps=6 , each step taking 5-10 seconds on the local model. One deny scenario took 55 seconds. The fix was catching the raise inside the scenario tool and returning the decision as a string, so the LLM sees it as a normal tool result instead of an error to retry. None of these show up in deterministic tests. They only surface when real framework code runs against real model endpoints. That's the live field test's job — and in v0.2.0, it did its job. Ship, then listen. The v0.1.0 article was the best thing that happened to v0.2.0. I wrote a PRD with 92 features before shipping. None of the 14 v0.2.0 features were in that PRD. They came from developers reading the article and saying "what about X?" The PRD was a planning document. The comments were a priority list grounded in real deployment experience. Next time, I'll ship sooner and plan less. The engine is the boring part — and that's good. It's a weighted sum across five dimensions with frozen band boundaries. It hasn't changed between v0.1.0 and v0.2.0, and it shouldn't. The interesting work is everything around it — what you validate before, what you redact after, what you observe across sessions, and how you serve it to non-Python fleets. A stable core with a growing surface is the right shape. Don't re-test the engine through the LLM. 2,490 deterministic assertions, zero LLM, 100% pass. The live test proves adapters, not engine correctness. The covering design 206 runs achieves the same coverage as the full 2,490-run cross-product at 12× reduction. The full cross-product is redundant because the engine is framework-agnostic — a scenario's decision depends only on scenario, agent class , never on langgraph vs crewai vs smolagents. not-available is the most misunderstood status in agent testing. It means "the LLM didn't call the right tool." It does NOT mean "the engine decided wrong." Distinguishing not-available from unexpected-decision is the difference between a flaky CI gate fails on model nondeterminism and a strict one fails only on real regressions . True in v0.1.0, confirmed across 4 models in v0.2.0. Next time, I'll track them separately from day one. Tier-1 multi-tool testing should be dropped from the release gate. It adds no engine coverage, demonstrates a known LLM limitation, and is the only source of flakiness. I spent hours trying to fix it — prompt strengthening made it worse , model swapping same result , individual retries nondeterministic . The fix isn't a better model or a better prompt. It's a different test design: one scenario per agent, which passes 100% on every model. Catch the raise, return the string. For deny/escalate in interactive frameworks, catching ToolTrustDecisionError and returning the decision as a string eliminates the retry loop. The LLM sees a result, not an error. The decision is still recorded. This one fix turned smolagents from a 55-second hang into a 5-second pass. Local for smoke, cloud for sweeps. The 55s vs 12s per-agent difference makes local impractical for full sweeps. Use local OMLX for quick checks, cloud glm-5 for the real run. glm-5 fixed 6 agents that local couldn't — but no model fixed tier-1. Hybrid is the answer. Reasoning models need token budget. gpt-oss-20b spends tokens on internal reasoning before emitting content. Send it max tokens=5 and it returns an empty assistant message. If your harness treats empty responses as failures, you'll debug your integration when the actual problem is token budget allocation. Model-id prefixes are non-negotiable connectivity. openai/{model} works on local OMLX. It stalls silently on OpenRouter — no timeout, no error, just a hang. You need openrouter/{model} for cloud. One line of code, one hour of debugging. pip install agent-tooltrust v0.2.0 1068+ tests. 91% coverage. Ruff clean. Mypy --strict clean. OWASP 10/10. Hardened baseline 15/15. 52 issues closed. No breaking API changes. not-available -style failures in their eval harness? How do you distinguish them from real regressions in CI? The gatekeeper became a control plane. 14 developers made it happen. Star the repo https://github.com/deghosal-2026/agent-tooltrust · Read the field test report https://github.com/deghosal-2026/agent-tooltrust/blob/main/docs/field-test/FIELD TEST REPORT-v0.2.0.md · pip install agent-tooltrust