You ship an AI agent to production. It runs smoothly for a week. Then one morning your LLM bill is 40% higher than expected—and you have no idea why. By the time you dig into logs, the damage is done.
The problem isn't that your agent broke loudly. It's that cost spikes are silent. Your agent succeeded. It returned tokens. It burned budget. And without a baseline to compare against, you won't see the spike until the bill arrives.
Every LLM agent has a natural token consumption pattern. For a given input, it tends to use roughly the same number of tokens each time—within a predictable range. That range is your baseline.
When an agent deviates sharply from its baseline, something changed:
The catch: a 30% spike in tokens is real and expensive, but it's invisible if you're only looking at raw counts. You need a baseline to know it's an anomaly at all.
The cleanest approach uses percentiles. Here's why percentiles work better than simple averages:
Median (p50): Half your runs use fewer tokens than this; half use more. It's stable and unaffected by outliers. If your agent usually takes 150 tokens per run and the p50 is 160, that's your normal.
95th percentile (p95): 95% of your runs fall below this threshold. It captures the "busy day" for your agent—the legitimate high-end of normal. If your p95 is 400 tokens, you expect some runs to hit that; they're not anomalies.
99th percentile (p99): The rare, expensive run. Useful for budgeting headroom, but too loose for early anomaly detection.
Why percentiles over averages? One runaway loop pushes an average up by 30%. The p95 barely moves—it's already accounting for variability. Averages lie when there are outliers. Percentiles don't.
Here's where most baselines fail: they're too trigger-happy. Alert on every spike above p95, and you'll get alerts when the agent just had a legitimately complex input. Ignore real spikes, and you miss the runaway loop that costs $500.
The trick is meaningful deviation: a spike that's both large and sustained.
Single-run spike above p99? Probably legitimate—the agent got a complex query, used more tokens, moved on. No alert needed yet.
Three runs in a row above p95, with at least one above p99? Now we're talking. Multiple high-cost runs suggest something systematic changed, not a one-off.
Weekly spend 50% above last week's baseline? That's a signal worth investigating.
The math is straightforward: if your agent normally costs $0.02 per run (p50 at 100 tokens * $0.00015/token for GPT-4o) and you see three runs at $0.08 each, that's 4x normal. Investigate.
Here's a concrete example. Say your agent summarizes customer feedback:
Run 1: 145 tokens
Run 2: 152 tokens
Run 3: 148 tokens
Run 4: 5,200 tokens ← anomaly
Run 5: 4,800 tokens ← anomaly again
Run 6: 156 tokens (back to normal)
Your p50 baseline: ~150 tokens. Your p95: ~160 tokens.
Runs 4 and 5 are 30–40x your baseline. One spike could be noise. Two in a row is a pattern. If you're running this agent 100 times a day, 30 of those runs suddenly costing 5000 tokens changes your bill overnight.
A baseline system flags this: "Runs 4–5 exceeded p99 + sustained above p95. Investigate: possible retry loop or tool recursion."
You check the logs, find the agent is calling a downstream API that's slow, triggering a retry loop. You add a timeout or a backoff. Crisis averted.
To build and track baselines, you need:
Most standard application monitoring tools (DataDog, New Relic, etc.) don't track model-specific token usage out of the box—you're usually exporting logs and computing percentiles yourself. That's labor-intensive and easy to miss.
A simpler approach: Use an observability tool built for LLM agents. The AI Agents Control Tower, for example, auto-computes p50/p95/p99 baselines per agent and alerts when a run spikes above them—no manual percentile math, no custom logging. You define your SLA (expected cost or latency), and the system flags deviations automatically.
The broader lesson: silent cost spikes are only silent if you have no baseline to compare against. The moment you establish what "normal" looks like for your agent—in tokens, cost, latency, or all three—anomalies become visible.
Before you ship your next AI agent:
Your bill (and your sleep) will thank you.
Have you seen a cost spike on an agent you didn't expect? What caught it—or what didn't?