cd /news/ai-agents/why-your-agent-burned-2800-at-3-am-a… · home topics ai-agents article
[ARTICLE · art-95332] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Why Your Agent Burned $2,800 at 3 AM — and How to See It

A developer built AgentShield, a pre-execution spend firewall for AI agents, after a runaway agent made 21 API calls costing $2,800 at 3 AM. The developer integrated it with Agent-Devtools, a causal debugger by Jacopo, to provide both spend blocking and post-hoc analysis of agent behavior. The integration uses a shared event schema and trace IDs to link spend evaluations with execution replays.

read3 min views1 publishedAug 13, 2026

Co-authored with Jacopo ( Agent-Devtools)

At 3:07 AM, my agent made 21 API calls to a premium LLM endpoint. Each cost $133. Total time: 60 seconds. Total bill: $2,800.

I was asleep. The budget alert email — the thing I'd set up so this couldn't happen — arrived 4 minutes later. Also while I was asleep. It was a very informative email about money that was already gone.

That night taught me something that took two open-source projects to fully solve:

You need a firewall to stop the bleed. You need a debugger to see why it happened.One without the other is half a safety system.

Before that night, I'd tried the standard defenses, in order:

So I built ** AgentShield**: a pre-execution spend firewall for AI agents. Every transaction is evaluated against your rules

APPROVED

, BLOCKED

, or FLAGGED

, in priority order, deterministically.The 3 AM incident wouldn't have survived contact with a single rule: transaction_limit: max_amount $100

.

That solves "what should we block right now?" It does not solve "why did it happen?"

Blocking a runaway agent is like unplugging a burning appliance. Safe. Necessary. But you still don't know if it was a bug in your code, a retry loop in a library, a poisoned tool response, or a bad prompt. Without that answer, the agent just runs again tomorrow — and you play firewall whack-a-mole.

That's the problem ** Agent-Devtools** — built by Jacopo — solves: a local-first causal debugger for agent runs. It answers "why did my agent behave this way?" with visual replay, behavior diffing, and full visibility into prompts, context, memory, retrieval, and tool calls — the exact execution timeline that led to the bad decision.

Jacopo found AgentShield through a comment on a LangChain issue about runaway agent loops, saw the same gap I did, and opened an issue: "You stop the bleed, I show the loop. Want to integrate?" We both said yes.

The integration is deliberately boring — a shared event schema, two small modules, no hosted services:

from agentshield import SpendControlEngine, SpendEvaluationEmitter
from agent_devtools import TraceStore
from agent_devtools.adapters.agentshield import make_agentshield_callback

store = TraceStore()
cb = make_agentshield_callback(store)

engine = SpendControlEngine()
emitter = SpendEvaluationEmitter(engine)

emitter.emit(
    transaction={"amount": "500.00", "merchant": "openai-api",
                 "category": "llm_inference"},
    rules=[{"id": "r1", "type": "transaction_limit", "priority": 1,
            "params": {"max_amount": "250.00"}, "action": "BLOCK"}],
    trace_id="trace_42",
    on_event=cb,
)

That's it. Under the hood:

agentshield.spend.evaluation

event per evaluation — NDJSON to stdout or a file (for tailing), or an in-process callback for embedding.trace_id

joins directly to Agent-Devtools' native run_id

, so spend decisions render triggered

, passed

, skipped

, or not_reached

. You see near-misses, not just the winning rule.The result is the loop the title promises: the firewall stops the spend at the moment it happens, and the debugger shows you the replay of why the agent got there.

The nice thing about this integration is how little ceremony it needed:

evaluate_with_trace()

  • SpendEvaluationEmitter

) — additive, so existing evaluate()

behavior didn't change.trace_id

crashing the host runtime (now falls back to unattributed

), callback events invisible in the run list (runs now auto-create), one malformed NDJSON line dropping the rest of a file (now skip-and-continue), and a Decimal precision bug on our side (now exact).Two repos, one issue thread, under a day from first comment to both sides merged.

Every agent team eventually has their own 3 AM. The only question is whether it's a $2,800 lesson or a $19/month non-event.

The pattern worth stealing isn't the code — it's the division of labor: pre-execution control answers "should this run?"; post-execution visibility answers "why did it run badly?" Monitoring tools only tell you what already happened. Budget alerts only tell you what already happened. An agent safety story needs both halves — and now they're one integration.

Both MIT. Both merged. Star both, wire them together, and sleep through the next 3 AM with confidence.

── more in #ai-agents 4 stories · sorted by recency
── more on @agentshield 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/why-your-agent-burne…] indexed:0 read:3min 2026-08-13 ·