{"slug": "your-agent-s-bill-jumped-40-it-never-errored-once", "title": "Your Agent's Bill Jumped 40%. It Never Errored Once.", "summary": "An engineer warns that AI agent cost spikes are often invisible until the bill arrives, and proposes using percentile-based baselines (p50, p95, p99) to detect anomalies early. The approach flags sustained deviations—such as three runs above p95 with one above p99—to catch runaway loops or retry errors before they inflate costs.", "body_md": "You ship an AI agent to production. It runs smoothly for a week. Then one morning your LLM bill is 40% higher than expected—and you have no idea why. By the time you dig into logs, the damage is done.\n\nThe problem isn't that your agent broke loudly. It's that cost spikes are *silent*. Your agent succeeded. It returned tokens. It burned budget. And without a baseline to compare against, you won't see the spike until the bill arrives.\n\nEvery LLM agent has a natural token consumption pattern. For a given input, it tends to use roughly the same number of tokens each time—within a predictable range. That range is your baseline.\n\nWhen an agent deviates sharply from its baseline, something changed:\n\nThe catch: a 30% spike in tokens is real and expensive, but it's *invisible* if you're only looking at raw counts. You need a baseline to know it's an anomaly at all.\n\nThe cleanest approach uses percentiles. Here's why percentiles work better than simple averages:\n\n**Median (p50):** Half your runs use fewer tokens than this; half use more. It's stable and unaffected by outliers. If your agent usually takes 150 tokens per run and the p50 is 160, that's your normal.\n\n**95th percentile (p95):** 95% of your runs fall below this threshold. It captures the \"busy day\" for your agent—the legitimate high-end of normal. If your p95 is 400 tokens, you expect some runs to hit that; they're not anomalies.\n\n**99th percentile (p99):** The rare, expensive run. Useful for budgeting headroom, but too loose for early anomaly detection.\n\n**Why percentiles over averages?** One runaway loop pushes an average up by 30%. The p95 barely moves—it's already accounting for variability. Averages lie when there are outliers. Percentiles don't.\n\nHere's where most baselines fail: they're too trigger-happy. Alert on every spike above p95, and you'll get alerts when the agent just had a legitimately complex input. Ignore real spikes, and you miss the runaway loop that costs $500.\n\nThe trick is **meaningful deviation**: a spike that's both *large* and *sustained*.\n\n**Single-run spike above p99?** Probably legitimate—the agent got a complex query, used more tokens, moved on. No alert needed yet.\n\n**Three runs in a row above p95, with at least one above p99?** Now we're talking. Multiple high-cost runs suggest something systematic changed, not a one-off.\n\n**Weekly spend 50% above last week's baseline?** That's a signal worth investigating.\n\nThe math is straightforward: if your agent normally costs $0.02 per run (p50 at 100 tokens * $0.00015/token for GPT-4o) and you see three runs at $0.08 each, that's 4x normal. Investigate.\n\nHere's a concrete example. Say your agent summarizes customer feedback:\n\n```\nRun 1: 145 tokens\nRun 2: 152 tokens\nRun 3: 148 tokens\nRun 4: 5,200 tokens ← anomaly\nRun 5: 4,800 tokens ← anomaly again\nRun 6: 156 tokens (back to normal)\n```\n\nYour p50 baseline: ~150 tokens. Your p95: ~160 tokens.\n\nRuns 4 and 5 are 30–40x your baseline. One spike could be noise. Two in a row is a pattern. If you're running this agent 100 times a day, 30 of those runs suddenly costing 5000 tokens changes your bill overnight.\n\nA baseline system flags this: \"Runs 4–5 exceeded p99 + sustained above p95. Investigate: possible retry loop or tool recursion.\"\n\nYou check the logs, find the agent is calling a downstream API that's slow, triggering a retry loop. You add a timeout or a backoff. Crisis averted.\n\nTo build and track baselines, you need:\n\nMost standard application monitoring tools (DataDog, New Relic, etc.) don't track model-specific token usage out of the box—you're usually exporting logs and computing percentiles yourself. That's labor-intensive and easy to miss.\n\n**A simpler approach:** Use an observability tool built for LLM agents. The AI Agents Control Tower, for example, auto-computes p50/p95/p99 baselines per agent and alerts when a run spikes above them—no manual percentile math, no custom logging. You define your SLA (expected cost or latency), and the system flags deviations automatically.\n\nThe broader lesson: **silent cost spikes are only silent if you have no baseline to compare against.** The moment you establish what \"normal\" looks like for your agent—in tokens, cost, latency, or all three—anomalies become visible.\n\nBefore you ship your next AI agent:\n\nYour bill (and your sleep) will thank you.\n\n*Have you seen a cost spike on an agent you didn't expect? What caught it—or what didn't?*", "url": "https://wpnews.pro/news/your-agent-s-bill-jumped-40-it-never-errored-once", "canonical_source": "https://dev.to/opsveritas/your-agents-bill-jumped-40-it-never-errored-once-34p2", "published_at": "2026-07-21 07:21:24+00:00", "updated_at": "2026-07-21 07:30:53.192358+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["DataDog", "New Relic", "AI Agents Control Tower", "GPT-4o"], "alternates": {"html": "https://wpnews.pro/news/your-agent-s-bill-jumped-40-it-never-errored-once", "markdown": "https://wpnews.pro/news/your-agent-s-bill-jumped-40-it-never-errored-once.md", "text": "https://wpnews.pro/news/your-agent-s-bill-jumped-40-it-never-errored-once.txt", "jsonld": "https://wpnews.pro/news/your-agent-s-bill-jumped-40-it-never-errored-once.jsonld"}}