{"slug": "why-are-we-using-a-sledgehammer-to-crack-a-nut", "title": "Why are we using a sledgehammer to crack a nut?", "summary": "A developer replaced a 7B parameter LLM running on a GPU with a 2.4 MB Go classifier for email triage, achieving sub-millisecond CPU latency and 100% reliability on deterministic rules. The three-layer filter uses deterministic rules first, then a small TF-IDF/logistic regression model, and only calls the LLM for uncertain cases. The author argues that most AI workflows should use the smallest possible tool and put the LLM last.", "body_md": "# Why are we using a sledgehammer to crack a nut?\n\nI recently audited my email triage agent. I was running a 7B parameter model (Qwen 2.5) via Ollama just to decide if an email was a \"newsletter\" or \"work.\" It was absurd. I had a GPU humming 24/7 and agonizingly slow inference just to identify a sender domain. I replaced that bloated mess with a tiny Go classifier that's only 2.4 MB.\n\nHere is the routing logic in Go:\n\nIf you're building an AI workflow, stop treating the LLM as the default. Use a practical tutorial approach: map your data, identify the obvious patterns, and use the smallest possible tool for the job. Most of the time, a basic classifier from a decade ago beats a modern LLM on efficiency and speed.\n\nThe strategy is simple: Put the LLM last.\n\n## The Triage Hierarchy\n\nInstead of a direct API call, I implemented a three-layer filter. If layer one can solve it, we stop. If not, we move to layer two. The LLM is the absolute last resort for the \"uncertain tail.\"\n\n**Layer 1: Deterministic Rules.** If the sender is\n\n, it's a newsletter. No brain cells required.[[email protected]](/cdn-cgi/l/email-protection)**Layer 2: Small ML Model.** Using TF-IDF and logistic regression on the CPU. Sub-millisecond latency.**Layer 3: The LLM.** Only triggered if the small model's confidence score is below a specific threshold.\n\nHere is the routing logic in Go:\n\n```\nfunc (c *Classifier) decide(m Message) Decision {\n\t// 1. Deterministic rules: obvious senders, decided at the door.\n\tif d := preClassify(m); d != nil {\n return *d\n\t}\n\t// 2. Small ML model: sub-millisecond, on CPU.\n\tpred := c.ml.Predict(m)\n\tif pred.Confidence >= c.threshold {\n return decisionFrom(pred)\n\t}\n\t// 3. LLM last: only the uncertain tail reaches the cloud.\n\treturn c.classifyWithLLM(m)\n}\n```\n\n## Performance Breakdown\n\nStop blindly chasing \"state-of-the-art\" and look at the actual cost-to-value ratio:\n\n**Resource Usage:** Swapped a power-hungry GPU for a few megabytes of RAM.**Latency:** Moved from seconds of waiting for an LLM response to sub-millisecond CPU execution.**Reliability:** A hard-coded rule is 100% predictable; a 7B model is a probabilistic guess.\n\nIf you're building an AI workflow, stop treating the LLM as the default. Use a practical tutorial approach: map your data, identify the obvious patterns, and use the smallest possible tool for the job. Most of the time, a basic classifier from a decade ago beats a modern LLM on efficiency and speed.\n\n[Next Building a Deep Learning Framework from Scratch →](/en/threads/2426/)\n\n## All Replies （3）\n\nA\n\nDid this with a small regex script for my filters and barely noticed a difference.\n\n0\n\nM\n\nTry a simple keyword filter first, usually handles 90% of the noise anyway.\n\n0\n\nJ\n\nSwitched to a tiny BERT model for similar tagging and it's basically instant now.\n\n0", "url": "https://wpnews.pro/news/why-are-we-using-a-sledgehammer-to-crack-a-nut", "canonical_source": "https://promptcube3.com/en/threads/2438/", "published_at": "2026-07-23 17:01:37+00:00", "updated_at": "2026-07-24 01:05:36.169641+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "developer-tools"], "entities": ["Qwen 2.5", "Ollama", "Go"], "alternates": {"html": "https://wpnews.pro/news/why-are-we-using-a-sledgehammer-to-crack-a-nut", "markdown": "https://wpnews.pro/news/why-are-we-using-a-sledgehammer-to-crack-a-nut.md", "text": "https://wpnews.pro/news/why-are-we-using-a-sledgehammer-to-crack-a-nut.txt", "jsonld": "https://wpnews.pro/news/why-are-we-using-a-sledgehammer-to-crack-a-nut.jsonld"}}