Why are we using a sledgehammer to crack a nut?

A developer replaced a 7B parameter LLM running on a GPU with a 2.4 MB Go classifier for email triage, achieving sub-millisecond CPU latency and 100% reliability on deterministic rules. The three-layer filter uses deterministic rules first, then a small TF-IDF/logistic regression model, and only calls the LLM for uncertain cases. The author argues that most AI workflows should use the smallest possible tool and put the LLM last.

Why are we using a sledgehammer to crack a nut? I recently audited my email triage agent. I was running a 7B parameter model Qwen 2.5 via Ollama just to decide if an email was a "newsletter" or "work." It was absurd. I had a GPU humming 24/7 and agonizingly slow inference just to identify a sender domain. I replaced that bloated mess with a tiny Go classifier that's only 2.4 MB. Here is the routing logic in Go: If you're building an AI workflow, stop treating the LLM as the default. Use a practical tutorial approach: map your data, identify the obvious patterns, and use the smallest possible tool for the job. Most of the time, a basic classifier from a decade ago beats a modern LLM on efficiency and speed. The strategy is simple: Put the LLM last. The Triage Hierarchy Instead of a direct API call, I implemented a three-layer filter. If layer one can solve it, we stop. If not, we move to layer two. The LLM is the absolute last resort for the "uncertain tail." Layer 1: Deterministic Rules. If the sender is , it's a newsletter. No brain cells required. email protected /cdn-cgi/l/email-protection Layer 2: Small ML Model. Using TF-IDF and logistic regression on the CPU. Sub-millisecond latency. Layer 3: The LLM. Only triggered if the small model's confidence score is below a specific threshold. Here is the routing logic in Go: func c Classifier decide m Message Decision { // 1. Deterministic rules: obvious senders, decided at the door. if d := preClassify m ; d = nil { return d } // 2. Small ML model: sub-millisecond, on CPU. pred := c.ml.Predict m if pred.Confidence = c.threshold { return decisionFrom pred } // 3. LLM last: only the uncertain tail reaches the cloud. return c.classifyWithLLM m } Performance Breakdown Stop blindly chasing "state-of-the-art" and look at the actual cost-to-value ratio: Resource Usage: Swapped a power-hungry GPU for a few megabytes of RAM. Latency: Moved from seconds of waiting for an LLM response to sub-millisecond CPU execution. Reliability: A hard-coded rule is 100% predictable; a 7B model is a probabilistic guess. If you're building an AI workflow, stop treating the LLM as the default. Use a practical tutorial approach: map your data, identify the obvious patterns, and use the smallest possible tool for the job. Most of the time, a basic classifier from a decade ago beats a modern LLM on efficiency and speed. Next Building a Deep Learning Framework from Scratch → /en/threads/2426/ All Replies （3） A Did this with a small regex script for my filters and barely noticed a difference. 0 M Try a simple keyword filter first, usually handles 90% of the noise anyway. 0 J Switched to a tiny BERT model for similar tagging and it's basically instant now. 0