How to Send 1% of Your Logs to an LLM and Still Catch Everything

A new architecture for log analysis using large language models reduces costs by feeding less than 1% of log lines to the model, using pattern mining and deterministic triage to filter out routine events before the LLM is called. The approach breaks the link between log volume and AI cost, scaling expenses with incidents rather than ingest.

Someone wires an LLM into the log pipeline, points it at production, and watches it work beautifully — for an afternoon. Then the bill arrives. A medium-sized service produces millions of log lines a day. Feed all of them to a model, and you’re looking at hundreds of dollars an hour, every hour, forever. The project gets quietly shut down by the end of the week, and the takeaway becomes “AI on logs is too expensive.” That conclusion is wrong. The architecture was wrong. The AI was put at the front of the pipeline, where it has to read everything, instead of at the end , where it only sees what nothing else could explain. The fix is an architecture, not a bigger budget or a cheaper model. Get it right, and the model sees less than 1% of your log lines — and the technique applies to any pipeline where an LLM call costs real money, not just log analysis. It’s the same triage pattern spam filters and fraud detection have used for decades, pointed at logs. The rest of this post is how you get to that number. To keep it concrete rather than hand-wavy, I’ll show real config from an open-source implementation. LLM cost is almost embarrassingly simple to model: cost ≈ tokens per call × calls × price per token You have three levers, and most teams only ever touch one. They shop for a cheaper price per token — a smaller model, a discount tier. That helps a little. But the lever that actually matters is the middle one: how many times you call the model at all. Here’s the trap. When the AI is the first thing in your pipeline, calls is tied to your log volume . Every line, or every batch of lines, is a call. Your bill scales with how much you log — which is to say, with your traffic, your debug-logging habits, and that one service that prints a stack trace on every request. It does not scale with how many things are actually wrong . That’s the real problem with cloud “AI observability” pricing, too: you pay for the firehose, not for the incidents. Two systems with identical reliability but different log verbosity get wildly different bills. That should feel broken, because it is. The fix is to break the link between “how much you log” and “how often you call the model.” You want your cost to scale with incidents , not ingest . The principle is one line: do the expensive thinking with cheap, deterministic code, and only spend tokens on the handful of events that genuinely need intelligence. Think of an emergency room. The specialist doesn’t examine everyone who walks in. A nurse checks vitals first. Normal? Go home. Off? Monitor. Dangerous? Now you get the specialist. The specialist — the expensive resource — sees a small fraction of arrivals, and every one of them is worth their time. Your LLM is the specialist. Everything before it is triage, and triage should be nearly free. Each stage before the model takes microseconds to milliseconds. Only the last box costs money. By the time a line reaches it, you already know it’s worth analyzing. That’s the whole trick. Four levers get you there. None of them involves the AI. Raw logs look unique even when they’re not. “Failed to connect to db 10.0.1.7 after 3 retries” and “Failed to connect to db 10.0.2.9 after 5 retries” are the same event with different variables. If you treat each line as unique, you drown. Pattern mining fixes this. A Drain-style algorithm strips the variable parts — IPs, IDs, numbers, timestamps — and clusters lines by their skeleton. A million lines collapse into a few dozen stable templates . Now “how often does this thing happen?” becomes a countable question, because “this thing” is a template, not a string. This is the foundation on which everything else stands. You can’t decide what’s normal until you can group like with like. php 2,300,000 raw lines/day -- pattern mining -- ~40 templates Forty things you can reason about, instead of two million you can’t. Once logs are grouped into templates, you learn each template’s normal rate during a training period. I wrote about how that training-to-live rollout works in “ Catching Errors Your Monitoring Doesn’t Know About https://medium.com/stackademic/catching-errors-you-dont-know-with-sre-agent-04962882102e ”. After training, every incoming event gets one of three verdicts: That first bucket is where the 99% goes. The vast majority of production logs are the same boring templates repeating: health checks, routine retries, the same deprecation warning since 2023. The model never needs to see them because deterministic code already knows they’re normal. This is the single highest-leverage decision in the whole design: known-and-normal is answered without an LLM. The AI is reserved for “new” and “abnormally frequent” — the two things cheap code genuinely can’t judge on its own. When something does get sent to the model, remember the answer. If the same template triggers again ten minutes later, reuse the finding instead of paying for a second call. The reason this works so well is Lever 1 again. Your cache key is the template ID , not the raw log text. So a hundred structurally-identical errors — different IPs, different retry counts, different user IDs — all hit the same cache entry. Without pattern mining, caching raw text rarely hits because the variables are always different. With it, the hit rate is high exactly when you need it: during an incident, when the same broken thing is screaming thousands of times a minute. Thirty minutes of “we already know about this” for the price of one call. Everything above reduces calls in the common case. This lever bounds the worst case. Give the pipeline a hard cap on model calls per hour. If something goes wrong upstream — a bad deploy changes every log format at once, so suddenly everything looks “unknown” — you do not want an unbounded spend. The cap is your circuit breaker. Past it, events wait. It feels crude, and that’s the point. You can reason about your maximum possible bill before it happens: max calls per hour × price per call × 24. A ceiling you can multiply on a napkin beats a system that's "usually cheap" and occasionally horrifying. Put the levers together on a realistic day: The model saw well under 1% of the firehose. Your cost now tracks incidents , not ingest — which is the whole thing you wanted. Stop measuring cost per log line. It’s the wrong denominator — it makes a chatty service look expensive when nothing is wrong. Measure cost per real incident . Watch two numbers: When those two are healthy, your spend is boring and predictable — which is exactly what you want a bill to be. If you build one thing from this, build the funnel, not the prompt. The AI isn’t the expensive part of an AI pipeline. Calling it too often is. Fix that, and “AI on our logs” stops being a line item someone kills on day two and becomes a cost you can defend to finance. If you want a working reference to read, the pattern miner, the verdicts, the cache, and the cap are all in an open-source implementation Versus Incident https://github.com/VersusControl/versus-incident . How to Send 1% of Your Logs to an LLM and Still Catch Everything https://pub.towardsai.net/how-to-send-1-of-your-logs-to-an-llm-and-still-catch-everything-dce28b1b5a93 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.