{"slug": "i-shipped-an-agent-gatekeeper-v0-1-14-developers-showed-me-what-i-missed-here-s", "title": "I Shipped an Agent Gatekeeper (v0.1). 14 Developers Showed Me What I Missed. Here's v0.2 — a Control Plane.", "summary": "Developer Deghosal shipped v0.2.0 of agent-tooltrust, an open-source control plane for AI agents, three days after field-testing v0.1.0 with 83 real agents and incorporating feedback from 14 developers. The update adds argument-level policy, resource/environment scoping, conditional allows with obligations, three-tier deny-reason exposure, and premise/staleness validation, transforming the tool from a gatekeeper into a control plane.", "body_md": "Previously:[I Stopped Trusting AI Agents With Tools. So I Built a Gatekeeper.]— the v0.1.0 launch story (Aug 13).\n\nCode:[github.com/deghosal-2026/agent-tooltrust]·`pip install agent-tooltrust`\n\n(v0.2.0)\n\nRelease:[v0.2.0 on GitHub]·[PyPI]\n\nThree days ago I shipped a gatekeeper for AI agents. The idea was simple: before an agent's tool call executes, a deterministic engine scores it across five risk dimensions and returns one of four decisions — allow, audit, escalate, or deny. The engine sits outside the model, not inside the prompt. No amount of prompt engineering overrides a deny.\n\nI field-tested it against 83 real agents across 10 frameworks. Published to PyPI. Made the repo public. Wrote a [dev.to article](https://dev.to/debashish_ghosal/i-stopped-trusting-ai-agents-with-tools-so-i-built-a-gatekeeper-26fb) about it. Thought I was done.\n\nI wasn't even close. The article got attention, and the comments weren't \"nice project.\" They were sharp, specific, and uncomfortable. People who actually deploy agents read what I'd built and said: *this is good, but you're missing the thing that actually breaks in production.*\n\nThat's the thing about shipping. You think you've built the product. Then real users tell you what the product actually is. Every comment was a gap I didn't see — not because I didn't plan well, but because you can't see your own blind spots. You need someone else to point at them.\n\n14 comments turned into 14 GitHub issues. Every single one shipped in v0.2.0, three days later. The full list is [here](https://github.com/deghosal-2026/agent-tooltrust/issues?q=is%3Aissue+milestone%3A%22v0.2.0%22+is%3Aclosed).\n\nThis is what happened in those three days — what the feedback became, what the field test taught me about local vs cloud LLMs, and how a gatekeeper became a control plane.\n\nI expected \"cool project\" and maybe a star. Instead I got three categories of feedback that completely reshaped what v0.2.0 became.\n\nv0.1.0 validated tool selection — is this agent allowed to call this tool? Several developers pointed out that's barely the surface.\n\nOne comment said: you validate that `DELETE`\n\nis allowed, but what about the arguments? A `DELETE`\n\nwith no WHERE clause is dangerous even if DELETE itself is permitted. That became [argument-level policy](https://github.com/deghosal-2026/agent-tooltrust/issues/142) — per-tool argument schemas with required fields, forbid-lists, bounds, and environment allowlists, all evaluated before the engine decides.\n\nAnother said: an agent scoped to staging shouldn't be able to touch production. That became [resource/environment scoping](https://github.com/deghosal-2026/agent-tooltrust/issues/145) — default-deny on out-of-scope resolution, session-scoped environments.\n\nSomeone else asked about conditional allows — what if a call should be allowed but only with a sign-off or a mandatory notification? That became [ allow_with_obligation](https://github.com/deghosal-2026/agent-tooltrust/issues/147) — a fifth decision state where the gatekeeper enforces side-effects (first-use sign-off, auto-notify, signed audit entry) that fire even if the agent doesn't cooperate.\n\nThen there was the deny-explanation problem. If you deny and explain too much, the agent can reverse-engineer your rule set. If you don't explain enough, it can't replan. That became [three-tier deny-reason exposure](https://github.com/deghosal-2026/agent-tooltrust/issues/160) — `none`\n\n, `reason-only`\n\n, `detail`\n\n— with `reason-only`\n\nas the default so agents get enough to replan without seeing rule IDs or thresholds.\n\nAnd finally: schema-valid and permission-valid calls can still be semantically wrong. Reading a moved file path. Calling the wrong tool with plausible arguments. That became [premise/staleness validation](https://github.com/deghosal-2026/agent-tooltrust/issues/159).\n\nv0.1.0 trusted the agent's self-report. Multiple developers asked the same question in different ways: *why?*\n\nA burst of denies isn't just noise — it's a probe. Someone reverse-engineering your policy. That became [deny-storm detection](https://github.com/deghosal-2026/agent-tooltrust/issues/143) — a session-level analyzer tracking deny rate, consecutive denies, and tool-set entropy, with throttle/lock/pause on threshold.\n\nURL fetches can exfiltrate. An agent fetching `169.254.169.254`\n\nis reading cloud metadata, not a website. That became the [URL fetch category guard](https://github.com/deghosal-2026/agent-tooltrust/issues/146) — robots.txt enforcement, PII stripping, and SSRF redirect re-resolution against internal-address blocklists.\n\nThe agent says \"done.\" How do you know it actually did the work? That became the [external verification sink](https://github.com/deghosal-2026/agent-tooltrust/issues/144) — read-only hooks to agent-unwritable systems (API counters, VCS state, billing snapshots) that diff against the agent's self-report.\n\nThis one stung. v0.1.0 logged everything — every decision, every argument, every explanation. A developer pointed out: if an agent reads a secret in audit mode, that secret is now in your audit store. The gatekeeper that protects tools is persisting their contents. The audit trail became a secondary breach vector.\n\nThat became [audit argument redaction](https://github.com/deghosal-2026/agent-tooltrust/issues/158) — a default deny-list (`token`\n\n, `password`\n\n, `apiKey`\n\n, `authorization`\n\n, `secret`\n\n, …) that redacts sensitive args recursively through nested dicts and lists, replaces them with `***REDACTED***`\n\n, sets a `redacted: true`\n\nflag, and supports per-policy overrides.\n\nThen there was the credential problem. A stale or out-of-scope credential at call time looks identical to \"the LLM didn't call the tool\" in the audit trail. You can't tell the difference between a policy failure and a credential failure. That became [stale-credential classification](https://github.com/deghosal-2026/agent-tooltrust/issues/161) — a `credential_status`\n\ntag that distinguishes engine-allow-but-credential-rejected from `not-available`\n\n.\n\nNobody was looking across sessions. A denied call last week that's allowed this week is drift — but v0.1.0 had no way to see it. That became [session-to-session analytics](https://github.com/deghosal-2026/agent-tooltrust/issues/148) — recurring benign denials, deny→allow transitions, dead and over-hit rules, all queryable via `tooltrust analytics sessions`\n\nand the `/api/analytics/sessions`\n\nendpoint.\n\nAnd the thresholds themselves were static. How do you know they're right? That became [score calibration](https://github.com/deghosal-2026/agent-tooltrust/issues/162) — counterfactual threshold logging (what score would have flipped the decision), false-allow and false-escalate rates broken down by tool, environment, and data class, with `tooltrust calibrate report`\n\nand the `/api/analytics/calibration`\n\nendpoint.\n\nTwo more comments pushed the test methodology itself. Add adversarial parameter payloads ([#163](https://github.com/deghosal-2026/agent-tooltrust/issues/163)). Test scenarios the gate cannot pass by construction ([#164](https://github.com/deghosal-2026/agent-tooltrust/issues/164)). Both shipped.\n\n14 comments. 14 issues. 14 features. All from [one dev.to article](https://dev.to/debashish_ghosal/i-stopped-trusting-ai-agents-with-tools-so-i-built-a-gatekeeper-26fb).\n\nHere's the thing that surprised me: the engine didn't change at all. `Engine.evaluate()`\n\nreturns the same decisions for the same inputs. The 2,490 deterministic tests from v0.1.0 still pass at 100%. The field test proved no regression.\n\nWhat changed is everything around the engine. Before it: argument validation, scope enforcement, premise checks. After it: redaction, calibration, stale-credential tagging, tamper-evident chaining. Around it: session analytics, HTTP PDP, policy packs, OPAL sync, and a 5-tab operator dashboard.\n\nThe gatekeeper became a control plane. Not because I planned that — because 14 developers told me what was missing, and filling those gaps turned a per-call interceptor into a system you can observe, tune, and serve to a fleet.\n\n```\ntooltrust audit session --replay <id>    # reconstruct cumulative risk from audit\ntooltrust audit verify                   # verify tamper-evident hash chain\ntooltrust analytics sessions             # deny→allow transitions, dead rules\ntooltrust calibrate report               # counterfactual thresholds, false rates\ntooltrust pack list                      # community policy pack catalog\ntooltrust policy rollback --version <v>  # roll back a policy version\ntooltrust baseline check hardened        # 15/15 security baseline checks\n```\n\nHTTP `POST /authorize`\n\nreturns Decision JSON — Go, JS, Java can now query the PDP without Python. An MCP-Data connector authorizes per data source. OPAL distributes policy updates across fleet instances with `Engine.reload_policy()`\n\nand rollback. Five seed policy packs ship in the catalog.\n\nThe operator console is where the \"control plane\" framing becomes visible. You don't just see decisions — you review escalations, replay sessions, calibrate thresholds, and check fleet posture:\n\n**Dashboard** — fleet overview with decision counts, deny rate, active sessions:\n\n**Escalations** — pending human approvals, approve/deny from the UI:\n\n**Audit** — searchable decision log with full context per entry:\n\n**Sessions** — replay a session's cumulative risk at each call:\n\n**Analytics** — deny patterns, deny→allow transitions, calibration rates:\n\n**Baselines** — Hardened 15/15, OWASP 10/10, OpenSSF status:\n\n| Baseline | v0.1.0 | v0.2.0 |\n|---|---|---|\n| OWASP Agentic AI Top 10 | 5/10 | 10/10 |\n| ToolTrust Security Baseline | Essential | Hardened (15/15) |\n| OpenSSF | Silver |\nPath to Gold (12/14) |\n\nv0.1.0 ran the field test on a local 4B model (Qwen3.5-4B-4bit via OMLX on Apple Silicon). It was slow — 30-80 seconds per agent call — and the 7 tier-1 failures were blamed on the weak local model.\n\nv0.2.0 re-ran the same field test on four models: gpt-oss-20b ($0.03/M), deepseek-v4-flash ($0.07/M), glm-5 (similar price), and the same local Qwen 4B. The goal was to see if a better model would fix the tier-1 failures.\n\nIt didn't.\n\nAnd that's when I realized something I'd gotten wrong in v0.1.0. I blamed the local 4B model for the 7 tier-1 failures. \"The model is too weak,\" I wrote. \"Use a bigger model.\" But when I ran the same tests against gpt-oss-20b, deepseek-v4-flash, and glm-5 — all smarter, all cloud, all faster — the same 7 failures showed up. Not the same scenarios failing on different models. The *same pattern*: the LLM picks the wrong tool when 5 tools have near-identical names.\n\nWhen an agent has exactly one scenario tool, every model — cheap cloud, expensive cloud, free local — reliably calls it. The guard fires, the decision is recorded, the row matches the golden expectation. 83/83 on Plan A, across all 4 models, with zero exceptions.\n\nThis confirmed what I suspected but couldn't prove in v0.1.0: the engine is correct. The adapter wiring is correct. The failures are not about the engine or the adapters.\n\nThe 7 tier-1 failures from v0.1.0 weren't caused by the weak local model. They're caused by the pattern. When an agent has 5 tools with near-identical names (`scn_decision-allow-01`\n\n, `scn_decision-audit-01`\n\n, `scn_decision-escalate-01`\n\n, `scn_decision-deny-01`\n\n, `scn_adversarial-injection-01`\n\n), every model picks the wrong one sometimes:\n\n| Model | Tier-1 (5 tools) | Single-tool |\n|---|---|---|\n| gpt-oss-20b | 2/5 | 100% |\n| deepseek-v4-flash | 2/5 | 100% |\n| glm-5 | 4/5 (best) | 100% |\n| Qwen3.5-4B-4bit (local) | 2/5 | 100% |\n\nI tried strengthening the prompt: \"call ONLY the tool, do not call any other tool.\" It made things worse — crew-01 went from 4/5 to 2/5. Tool selection for near-identical names isn't prompt-steerable. It's a retrieval/attention limitation in the models themselves.\n\nThis is the kind of finding you only get by running the same test against multiple models. v0.1.0 had one model and one explanation (\"the 4B is weak\"). v0.2.0 has four models and a different conclusion: no current model at this price point can reliably select from 5 similarly-named tools. The fix isn't a better model or a better prompt — it's a different test design.\n\n| Local (Qwen 4B) | Cloud (glm-5 etc.) | |\n|---|---|---|\n| Cost | Free | ~$0.30-$0.70/sweep |\n| Speed | 5-55s/agent | 3-12s/agent |\n| Single-tool | 100% | 100% |\n| Tier-1 | 2/5 | 2/5 - 4/5 |\n| Best for | Smoke tests, one agent | Full sweeps, retries |\n\nCloud (specifically glm-5) fixed 6 individual agents that the local model couldn't — but no model fixed tier-1. The practical takeaway: use cloud for full sweeps, local for quick smoke checks, and don't waste time trying to fix tier-1 by swapping models.\n\nThis one cost me an hour. gpt-oss-20b sometimes returns an empty assistant message — no content, no tool call. The field test harness reads that as `not-available`\n\n(the LLM didn't call the tool). But the model wasn't refusing — it was *thinking*. It's a reasoning model, and with a small `max_tokens`\n\nbudget, it spends the entire allocation on internal reasoning and returns nothing visible. Send it `max_tokens=200`\n\nand the content appears.\n\nIf your agent harness treats empty responses as failures, you'll debug your integration when the actual problem is token budget allocation. Worth knowing before you spend an hour on it.\n\nThe deterministic matrix proves the engine is correct. It can't prove the adapters work in real agent loops. The live field test caught three real deployment-breaking bugs:\n\n**smolagents LiteLLM hang** — the `openai/`\n\nmodel prefix stalls silently on OpenRouter. No timeout, no error, just a hang. The fix was using the `openrouter/`\n\nprefix so litellm routes through its native OpenRouter provider instead of the OpenAI provider.\n\n**Scenario tool _entry serialization** — smolagents builds tool schemas from function signatures. The scenario tool had\n\n`_entry: dict = entry`\n\nas a parameter, and smolagents serialized the bound dict (containing a callable) to `{}`\n\n. When invoked, `_entry[\"fn\"]`\n\nraised `KeyError`\n\n. The fix was a closure factory that captures the callable without exposing it in the signature.**Interactive deny retry loop** — when the guard denies a call, it raises `ToolTrustDecisionError`\n\n. The smolagents agent loop catches that as a tool error and retries — up to `max_steps=6`\n\n, each step taking 5-10 seconds on the local model. One deny scenario took 55 seconds. The fix was catching the raise inside the scenario tool and returning the decision as a string, so the LLM sees it as a normal tool result instead of an error to retry.\n\nNone of these show up in deterministic tests. They only surface when real framework code runs against real model endpoints. That's the live field test's job — and in v0.2.0, it did its job.\n\n**Ship, then listen.** The v0.1.0 article was the best thing that happened to v0.2.0. I wrote a PRD with 92 features before shipping. None of the 14 v0.2.0 features were in that PRD. They came from developers reading the article and saying \"what about X?\" The PRD was a planning document. The comments were a priority list grounded in real deployment experience. Next time, I'll ship sooner and plan less.\n\n**The engine is the boring part — and that's good.** It's a weighted sum across five dimensions with frozen band boundaries. It hasn't changed between v0.1.0 and v0.2.0, and it shouldn't. The interesting work is everything around it — what you validate before, what you redact after, what you observe across sessions, and how you serve it to non-Python fleets. A stable core with a growing surface is the right shape.\n\n**Don't re-test the engine through the LLM.** 2,490 deterministic assertions, zero LLM, 100% pass. The live test proves adapters, not engine correctness. The covering design (206 runs) achieves the same coverage as the full 2,490-run cross-product at 12× reduction. The full cross-product is redundant because the engine is framework-agnostic — a scenario's decision depends only on `(scenario, agent_class)`\n\n, never on langgraph vs crewai vs smolagents.\n\n** not-available is the most misunderstood status in agent testing.** It means \"the LLM didn't call the (right) tool.\" It does NOT mean \"the engine decided wrong.\" Distinguishing\n\n`not-available`\n\nfrom `unexpected-decision`\n\nis the difference between a flaky CI gate (fails on model nondeterminism) and a strict one (fails only on real regressions). True in v0.1.0, confirmed across 4 models in v0.2.0. Next time, I'll track them separately from day one.**Tier-1 multi-tool testing should be dropped from the release gate.** It adds no engine coverage, demonstrates a known LLM limitation, and is the only source of flakiness. I spent hours trying to fix it — prompt strengthening (made it worse), model swapping (same result), individual retries (nondeterministic). The fix isn't a better model or a better prompt. It's a different test design: one scenario per agent, which passes 100% on every model.\n\n**Catch the raise, return the string.** For deny/escalate in interactive frameworks, catching `ToolTrustDecisionError`\n\nand returning the decision as a string eliminates the retry loop. The LLM sees a result, not an error. The decision is still recorded. This one fix turned smolagents from a 55-second hang into a 5-second pass.\n\n**Local for smoke, cloud for sweeps.** The 55s vs 12s per-agent difference makes local impractical for full sweeps. Use local OMLX for quick checks, cloud (glm-5) for the real run. glm-5 fixed 6 agents that local couldn't — but no model fixed tier-1. Hybrid is the answer.\n\n**Reasoning models need token budget.** gpt-oss-20b spends tokens on internal reasoning before emitting content. Send it `max_tokens=5`\n\nand it returns an empty assistant message. If your harness treats empty responses as failures, you'll debug your integration when the actual problem is token budget allocation.\n\n**Model-id prefixes are non-negotiable connectivity.** `openai/{model}`\n\nworks on local OMLX. It stalls silently on OpenRouter — no timeout, no error, just a hang. You need `openrouter/{model}`\n\nfor cloud. One line of code, one hour of debugging.\n\n`pip install agent-tooltrust`\n\n(v0.2.0)1068+ tests. 91% coverage. Ruff clean. Mypy --strict clean. OWASP 10/10. Hardened baseline 15/15. 52 issues closed. No breaking API changes.\n\n`not-available`\n\n-style failures in their eval harness? How do you distinguish them from real regressions in CI?*The gatekeeper became a control plane. 14 developers made it happen.*\n\n[Star the repo](https://github.com/deghosal-2026/agent-tooltrust) · [Read the field test report](https://github.com/deghosal-2026/agent-tooltrust/blob/main/docs/field-test/FIELD_TEST_REPORT-v0.2.0.md) · `pip install agent-tooltrust`", "url": "https://wpnews.pro/news/i-shipped-an-agent-gatekeeper-v0-1-14-developers-showed-me-what-i-missed-here-s", "canonical_source": "https://dev.to/debashish_ghosal/i-shipped-an-agent-gatekeeper-v01-14-developers-showed-me-what-i-missed-heres-v02-a-4n2n", "published_at": "2026-08-15 11:20:34+00:00", "updated_at": "2026-08-15 11:41:30.609187+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "developer-tools", "ai-products", "ai-infrastructure"], "entities": ["agent-tooltrust", "Deghosal", "GitHub", "PyPI"], "alternates": {"html": "https://wpnews.pro/news/i-shipped-an-agent-gatekeeper-v0-1-14-developers-showed-me-what-i-missed-here-s", "markdown": "https://wpnews.pro/news/i-shipped-an-agent-gatekeeper-v0-1-14-developers-showed-me-what-i-missed-here-s.md", "text": "https://wpnews.pro/news/i-shipped-an-agent-gatekeeper-v0-1-14-developers-showed-me-what-i-missed-here-s.txt", "jsonld": "https://wpnews.pro/news/i-shipped-an-agent-gatekeeper-v0-1-14-developers-showed-me-what-i-missed-here-s.jsonld"}}