I recently audited my email triage agent. I was running a 7B parameter model (Qwen 2.5) via Ollama just to decide if an email was a "newsletter" or "work." It was absurd. I had a GPU humming 24/7 and agonizingly slow inference just to identify a sender domain. I replaced that bloated mess with a tiny Go classifier that's only 2.4 MB.
Here is the routing logic in Go:
If you're building an AI workflow, stop treating the LLM as the default. Use a practical tutorial approach: map your data, identify the obvious patterns, and use the smallest possible tool for the job. Most of the time, a basic classifier from a decade ago beats a modern LLM on efficiency and speed.
The strategy is simple: Put the LLM last.
The Triage Hierarchy #
Instead of a direct API call, I implemented a three-layer filter. If layer one can solve it, we stop. If not, we move to layer two. The LLM is the absolute last resort for the "uncertain tail."
Layer 1: Deterministic Rules. If the sender is
, it's a newsletter. No brain cells required.[email protected]Layer 2: Small ML Model. Using TF-IDF and logistic regression on the CPU. Sub-millisecond latency.Layer 3: The LLM. Only triggered if the small model's confidence score is below a specific threshold.
Here is the routing logic in Go:
func (c *Classifier) decide(m Message) Decision {
// 1. Deterministic rules: obvious senders, decided at the door.
if d := preClassify(m); d != nil {
return *d
}
// 2. Small ML model: sub-millisecond, on CPU.
pred := c.ml.Predict(m)
if pred.Confidence >= c.threshold {
return decisionFrom(pred)
}
// 3. LLM last: only the uncertain tail reaches the cloud.
return c.classifyWithLLM(m)
}
Performance Breakdown #
Stop blindly chasing "state-of-the-art" and look at the actual cost-to-value ratio:
Resource Usage: Swapped a power-hungry GPU for a few megabytes of RAM.Latency: Moved from seconds of waiting for an LLM response to sub-millisecond CPU execution.Reliability: A hard-coded rule is 100% predictable; a 7B model is a probabilistic guess.
If you're building an AI workflow, stop treating the LLM as the default. Use a practical tutorial approach: map your data, identify the obvious patterns, and use the smallest possible tool for the job. Most of the time, a basic classifier from a decade ago beats a modern LLM on efficiency and speed.
Next Building a Deep Learning Framework from Scratch →
All Replies （3） #
A
Did this with a small regex script for my filters and barely noticed a difference.
0
M
Try a simple keyword filter first, usually handles 90% of the noise anyway.
0
J
Switched to a tiny BERT model for similar tagging and it's basically instant now.
0