Your LLM is probably overkill for text classification

A benchmark run on a MacBook Air M2 with 16GB RAM found that a frozen BGE-large sentence embedding model with a logistic regression head, packaged as the jeffy-classify tool, reached 94.3% accuracy at 118ms and $0 cost per 1,000 requests on 100 held-out Banking77 customer-intent samples, while local LLMs run through Ollama at temperature 0 scored between 46% and 68%. The best LLM in the test, phi4 (14B), hit 68% accuracy at 15.4 seconds and roughly $0.30 per 1,000 requests, and gemma4 (9.6B) also reached 68% at 696ms and about $0.15. The author argues that for multi-class text classification such as support ticket routing and intent detection, training a classifier on frozen embeddings avoids LLM latency, cost and malformed JSON output.

I keep seeing the same pattern. Someone needs to classify text route support tickets, tag documents, detect intent so they prompt an LLM. It works in the demo. A few months later the complaints are about latency, cost, and malformed JSON breaking things downstream There's a simpler approach that gets better results on classification: freeze a sentence embedding model and train a logistic regression head on top. That's it. No fine-tuning, no GPU, no prompt engineering I'll show both approaches on the same task so you can see the difference Banking77 is a customer intent dataset with 77 classes. Things like "lost or stolen card", "card payment not recognised", "contactless not working". Many labels are semantically close, which is what makes it a real test This is the kind of problem that shows up everywhere: support ticket routing, email classification, voice assistant intent detection. Lots of categories, subtle differences between them Here's how you'd classify text with an LLM. Using OpenAI as the example since that's what most people reach for: python LLM classification with OpenAI import openai import json client = openai.OpenAI labels = "activate my card", "age limit", "apple pay or google pay", "atm support", "automatic top up", "balance not updated after bank transfer", "balance not updated after cheque or cash deposit", "beneficiary not allowed", "cancel transfer", "card about to expire", "card acceptance", ... 66 more labels ... "virtual card not working", "visa or mastercard", "why verify identity", "wrong amount of cash received", "wrong exchange rate for cash withdrawal" def classify with llm text : response = client.chat.completions.create model="gpt-4o-mini", temperature=0, messages= { "role": "user", "content": f"""Classify this customer message into exactly one of these categories: {json.dumps labels } Message: {text} Reply with ONLY the category name, nothing else.""" } return response.choices 0 .message.content.strip result = classify with llm "I was charged twice for the same transaction" → "card payment not recognised" maybe, if the JSON doesn't break Problems with this: Here's the same task with Jeffy: Classification with frozen embeddings + logistic regression pip install jeffy-classify python from jeffy import Engine engine = Engine result = engine.predict "banking77", "I was charged twice for the same transaction" print result "label" → "card payment not recognised" print result "confidence" → 0.72 print result "latency ms" → 118 Three lines. No prompt, no API key, no output parsing. The embedding model BGE-large runs locally via ONNX, and the classifier is logistic regression coefficients trained on the standard banking77 training set. The whole thing runs on CPU client = openai.OpenAI labels = "activate my card", "age limit", ... 77 labels response = client.chat .completions.create model="gpt-4o-mini", messages= { "role": "user", "content": f"Classify: ..." } label = response.choices 0 .message.content.strip hope it's valid python from jeffy import Engine engine = Engine result = engine.predict "banking77", "I was charged twice" label = result "label" score = result "confidence" always structured JSON I ran this benchmark on a MacBook Air M2 with 16GB RAM. All LLMs via Ollama at temperature 0. 100 held-out test samples | Model | Type | Accuracy | Latency | Cost/1K | |---|---|---|---|---| | Jeffy BGE + LR | Embeddings | 94.3% | 118ms | $0 | | phi4 14B | LLM | 68% | 15.4s | ~$0.30 | | gemma4 9.6B | LLM | 68% | 696ms | ~$0.15 | | gemma3 4B | LLM | 64% | 956ms | ~$0.08 | | llama3.1 8B | LLM | 62% | 8.9s | ~$0.12 | | phi4-mini 3.8B | LLM | 52% | 181ms | ~$0.05 | | llama3.2 3B | LLM | 46% | 143ms | ~$0.04 | | mistral 7B | LLM | 39% | 17s | ~$0.12 | | phi 2.7B | LLM | 30% | 231ms | ~$0.04 | | qwen2.5 0.5B | LLM | 15% | 55ms | ~$0.01 | LLMs are generalists. They weren't trained specifically on banking intent labels. When you give them 77 options like "card not working", "contactless not working", and "card payment not recognised", they have to figure out the difference from the label names alone A trained classifier saw labeled examples during training. It learned the embedding-space boundaries between "lost my card" and "card won't tap" and "charged twice." It's not guessing from label names. It knows This matters more as class count grows. At 2 classes positive/negative sentiment , LLMs do fine. By 77 classes, they collapse Hosted LLMs are better than local ones at this, sure. But the fundamental problem stays: you're parsing generated text, paying per token, sending data to an external API, and dealing with seconds of latency per call. A recent paper "The Embedder's Dilemma", COLM 2026 https://arxiv.org/abs/2608.12875 benchmarked 10 LLMs against 26 embedding models across 37 tasks and found embedding models beat LLMs on classification specifically, with LLMs costing up to 1,431x more If you don't have a pretrained classifier for your task, training one takes seconds: python Train a custom classifier from jeffy import Engine engine = Engine texts = "reset my password", "can't log in", "upgrade my plan", "cancel subscription", ... labels = "account access", "account access", "billing", "billing", ... engine.train "my support task", texts, labels result = engine.predict "my support task", "I forgot my login" → {"label": "account access", "confidence": 0.91} The logistic regression head retrains in seconds on CPU. No GPU, no fine-tuning loop, no hyperparameter search. You can retrain it as new labeled data comes in pip install jeffy-classify jeffy-serve Open localhost:8400 and you have 68 pretrained classifiers running locally. Or try them online at playground.jeffyclassify.com https://playground.jeffyclassify.com