{"slug": "why-your-agent-burned-2800-at-3-am-and-how-to-see-it", "title": "Why Your Agent Burned $2,800 at 3 AM — and How to See It", "summary": "A developer built AgentShield, a pre-execution spend firewall for AI agents, after a runaway agent made 21 API calls costing $2,800 at 3 AM. The developer integrated it with Agent-Devtools, a causal debugger by Jacopo, to provide both spend blocking and post-hoc analysis of agent behavior. The integration uses a shared event schema and trace IDs to link spend evaluations with execution replays.", "body_md": "*Co-authored with Jacopo ( Agent-Devtools)*\n\nAt 3:07 AM, my agent made 21 API calls to a premium LLM endpoint. Each cost $133. Total time: 60 seconds. Total bill: **$2,800**.\n\nI was asleep. The budget alert email — the thing I'd set up so this couldn't happen — arrived 4 minutes later. Also while I was asleep. It was a very informative email about money that was already gone.\n\nThat night taught me something that took two open-source projects to fully solve:\n\nYou need a firewall to stop the bleed. You need a debugger to see why it happened.One without the other is half a safety system.\n\nBefore that night, I'd tried the standard defenses, in order:\n\nSo I built ** AgentShield**: a pre-execution spend firewall for AI agents. Every transaction is evaluated against your rules\n\n`APPROVED`\n\n, `BLOCKED`\n\n, or `FLAGGED`\n\n, in priority order, deterministically.The 3 AM incident wouldn't have survived contact with a single rule: `transaction_limit: max_amount $100`\n\n.\n\nThat solves *\"what should we block right now?\"* It does not solve *\"why did it happen?\"*\n\nBlocking a runaway agent is like unplugging a burning appliance. Safe. Necessary. But you still don't know if it was a bug in your code, a retry loop in a library, a poisoned tool response, or a bad prompt. Without that answer, the agent just runs again tomorrow — and you play firewall whack-a-mole.\n\nThat's the problem ** Agent-Devtools** — built by Jacopo — solves: a local-first causal debugger for agent runs. It answers \"why did my agent behave this way?\" with visual replay, behavior diffing, and full visibility into prompts, context, memory, retrieval, and tool calls — the exact execution timeline that led to the bad decision.\n\nJacopo found AgentShield through a comment on a LangChain issue about runaway agent loops, saw the same gap I did, and opened an issue: *\"You stop the bleed, I show the loop. Want to integrate?\"* We both said yes.\n\nThe integration is deliberately boring — a shared event schema, two small modules, no hosted services:\n\n``` python\nfrom agentshield import SpendControlEngine, SpendEvaluationEmitter\nfrom agent_devtools import TraceStore\nfrom agent_devtools.adapters.agentshield import make_agentshield_callback\n\nstore = TraceStore()\ncb = make_agentshield_callback(store)\n\nengine = SpendControlEngine()\nemitter = SpendEvaluationEmitter(engine)\n\n# Every evaluation flows into your trace store automatically\nemitter.emit(\n    transaction={\"amount\": \"500.00\", \"merchant\": \"openai-api\",\n                 \"category\": \"llm_inference\"},\n    rules=[{\"id\": \"r1\", \"type\": \"transaction_limit\", \"priority\": 1,\n            \"params\": {\"max_amount\": \"250.00\"}, \"action\": \"BLOCK\"}],\n    trace_id=\"trace_42\",\n    on_event=cb,\n)\n```\n\nThat's it. Under the hood:\n\n`agentshield.spend.evaluation`\n\nevent per evaluation — NDJSON to stdout or a file (for tailing), or an in-process callback for embedding.`trace_id`\n\njoins directly to Agent-Devtools' native `run_id`\n\n, so spend decisions render `triggered`\n\n, `passed`\n\n, `skipped`\n\n, or `not_reached`\n\n. You see near-misses, not just the winning rule.The result is the loop the title promises: **the firewall stops the spend at the moment it happens, and the debugger shows you the replay of why the agent got there.**\n\nThe nice thing about this integration is how little ceremony it needed:\n\n`evaluate_with_trace()`\n\n+ `SpendEvaluationEmitter`\n\n) — additive, so existing `evaluate()`\n\nbehavior didn't change.`trace_id`\n\ncrashing the host runtime (now falls back to `unattributed`\n\n), callback events invisible in the run list (runs now auto-create), one malformed NDJSON line dropping the rest of a file (now skip-and-continue), and a Decimal precision bug on our side (now exact).Two repos, one issue thread, under a day from first comment to both sides merged.\n\nEvery agent team eventually has their own 3 AM. The only question is whether it's a $2,800 lesson or a $19/month non-event.\n\nThe pattern worth stealing isn't the code — it's the division of labor: **pre-execution control answers \"should this run?\"; post-execution visibility answers \"why did it run badly?\"** Monitoring tools only tell you what already happened. Budget alerts only tell you what already happened. An agent safety story needs both halves — and now they're one integration.\n\nBoth MIT. Both merged. Star both, wire them together, and sleep through the next 3 AM with confidence.", "url": "https://wpnews.pro/news/why-your-agent-burned-2800-at-3-am-and-how-to-see-it", "canonical_source": "https://dev.to/maryan_k_bef6cf83fa64e809/why-your-agent-burned-2800-at-3-am-and-how-to-see-it-21l1", "published_at": "2026-08-13 13:36:02+00:00", "updated_at": "2026-08-13 13:50:53.926341+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "developer-tools", "ai-infrastructure"], "entities": ["AgentShield", "Agent-Devtools", "Jacopo", "LangChain", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/why-your-agent-burned-2800-at-3-am-and-how-to-see-it", "markdown": "https://wpnews.pro/news/why-your-agent-burned-2800-at-3-am-and-how-to-see-it.md", "text": "https://wpnews.pro/news/why-your-agent-burned-2800-at-3-am-and-how-to-see-it.txt", "jsonld": "https://wpnews.pro/news/why-your-agent-burned-2800-at-3-am-and-how-to-see-it.jsonld"}}