Co-authored with Jacopo ( Agent-Devtools)
At 3:07 AM, my agent made 21 API calls to a premium LLM endpoint. Each cost $133. Total time: 60 seconds. Total bill: $2,800.
I was asleep. The budget alert email — the thing I'd set up so this couldn't happen — arrived 4 minutes later. Also while I was asleep. It was a very informative email about money that was already gone.
That night taught me something that took two open-source projects to fully solve:
You need a firewall to stop the bleed. You need a debugger to see why it happened.One without the other is half a safety system.
Before that night, I'd tried the standard defenses, in order:
So I built ** AgentShield**: a pre-execution spend firewall for AI agents. Every transaction is evaluated against your rules
APPROVED
, BLOCKED
, or FLAGGED
, in priority order, deterministically.The 3 AM incident wouldn't have survived contact with a single rule: transaction_limit: max_amount $100
.
That solves "what should we block right now?" It does not solve "why did it happen?"
Blocking a runaway agent is like unplugging a burning appliance. Safe. Necessary. But you still don't know if it was a bug in your code, a retry loop in a library, a poisoned tool response, or a bad prompt. Without that answer, the agent just runs again tomorrow — and you play firewall whack-a-mole.
That's the problem ** Agent-Devtools** — built by Jacopo — solves: a local-first causal debugger for agent runs. It answers "why did my agent behave this way?" with visual replay, behavior diffing, and full visibility into prompts, context, memory, retrieval, and tool calls — the exact execution timeline that led to the bad decision.
Jacopo found AgentShield through a comment on a LangChain issue about runaway agent loops, saw the same gap I did, and opened an issue: "You stop the bleed, I show the loop. Want to integrate?" We both said yes.
The integration is deliberately boring — a shared event schema, two small modules, no hosted services:
from agentshield import SpendControlEngine, SpendEvaluationEmitter
from agent_devtools import TraceStore
from agent_devtools.adapters.agentshield import make_agentshield_callback
store = TraceStore()
cb = make_agentshield_callback(store)
engine = SpendControlEngine()
emitter = SpendEvaluationEmitter(engine)
emitter.emit(
transaction={"amount": "500.00", "merchant": "openai-api",
"category": "llm_inference"},
rules=[{"id": "r1", "type": "transaction_limit", "priority": 1,
"params": {"max_amount": "250.00"}, "action": "BLOCK"}],
trace_id="trace_42",
on_event=cb,
)
That's it. Under the hood:
agentshield.spend.evaluation
event per evaluation — NDJSON to stdout or a file (for tailing), or an in-process callback for embedding.trace_id
joins directly to Agent-Devtools' native run_id
, so spend decisions render triggered
, passed
, skipped
, or not_reached
. You see near-misses, not just the winning rule.The result is the loop the title promises: the firewall stops the spend at the moment it happens, and the debugger shows you the replay of why the agent got there.
The nice thing about this integration is how little ceremony it needed:
evaluate_with_trace()
SpendEvaluationEmitter
) — additive, so existing evaluate()
behavior didn't change.trace_id
crashing the host runtime (now falls back to unattributed
), callback events invisible in the run list (runs now auto-create), one malformed NDJSON line dropping the rest of a file (now skip-and-continue), and a Decimal precision bug on our side (now exact).Two repos, one issue thread, under a day from first comment to both sides merged.
Every agent team eventually has their own 3 AM. The only question is whether it's a $2,800 lesson or a $19/month non-event.
The pattern worth stealing isn't the code — it's the division of labor: pre-execution control answers "should this run?"; post-execution visibility answers "why did it run badly?" Monitoring tools only tell you what already happened. Budget alerts only tell you what already happened. An agent safety story needs both halves — and now they're one integration.
Both MIT. Both merged. Star both, wire them together, and sleep through the next 3 AM with confidence.