{"slug": "your-llm-is-probably-overkill-for-text-classification", "title": "Your LLM is probably overkill for text classification", "summary": "A benchmark run on a MacBook Air M2 with 16GB RAM found that a frozen BGE-large sentence embedding model with a logistic regression head, packaged as the jeffy-classify tool, reached 94.3% accuracy at 118ms and $0 cost per 1,000 requests on 100 held-out Banking77 customer-intent samples, while local LLMs run through Ollama at temperature 0 scored between 46% and 68%. The best LLM in the test, phi4 (14B), hit 68% accuracy at 15.4 seconds and roughly $0.30 per 1,000 requests, and gemma4 (9.6B) also reached 68% at 696ms and about $0.15. The author argues that for multi-class text classification such as support ticket routing and intent detection, training a classifier on frozen embeddings avoids LLM latency, cost and malformed JSON output.", "body_md": "I keep seeing the same pattern. Someone needs to classify text (route support tickets, tag documents, detect intent) so they prompt an LLM. It works in the demo. A few months later the complaints are about latency, cost, and malformed JSON breaking things downstream\n\nThere's a simpler approach that gets better results on classification: freeze a sentence embedding model and train a logistic regression head on top. That's it. No fine-tuning, no GPU, no prompt engineering\n\nI'll show both approaches on the same task so you can see the difference\n\nBanking77 is a customer intent dataset with 77 classes. Things like \"lost_or_stolen_card\", \"card_payment_not_recognised\", \"contactless_not_working\". Many labels are semantically close, which is what makes it a real test\n\nThis is the kind of problem that shows up everywhere: support ticket routing, email classification, voice assistant intent detection. Lots of categories, subtle differences between them\n\nHere's how you'd classify text with an LLM. Using OpenAI as the example since that's what most people reach for:\n\n``` python\n# LLM classification with OpenAI\nimport openai\nimport json\n\nclient = openai.OpenAI()\n\nlabels = [\n    \"activate_my_card\", \"age_limit\", \"apple_pay_or_google_pay\",\n    \"atm_support\", \"automatic_top_up\", \"balance_not_updated_after_bank_transfer\",\n    \"balance_not_updated_after_cheque_or_cash_deposit\", \"beneficiary_not_allowed\",\n    \"cancel_transfer\", \"card_about_to_expire\", \"card_acceptance\",\n    # ... 66 more labels ...\n    \"virtual_card_not_working\", \"visa_or_mastercard\",\n    \"why_verify_identity\", \"wrong_amount_of_cash_received\",\n    \"wrong_exchange_rate_for_cash_withdrawal\"\n]\n\ndef classify_with_llm(text):\n    response = client.chat.completions.create(\n        model=\"gpt-4o-mini\",\n        temperature=0,\n        messages=[{\n            \"role\": \"user\",\n            \"content\": f\"\"\"Classify this customer message into exactly\none of these categories: {json.dumps(labels)}\n\nMessage: {text}\n\nReply with ONLY the category name, nothing else.\"\"\"\n        }]\n    )\n    return response.choices[0].message.content.strip()\n\nresult = classify_with_llm(\"I was charged twice for the same transaction\")\n# → \"card_payment_not_recognised\" (maybe, if the JSON doesn't break)\n```\n\nProblems with this:\n\nHere's the same task with Jeffy:\n\n```\n# Classification with frozen embeddings + logistic regression\npip install jeffy-classify\npython\nfrom jeffy import Engine\n\nengine = Engine()\nresult = engine.predict(\"banking77\", \"I was charged twice for the same transaction\")\n\nprint(result[\"label\"])       # → \"card_payment_not_recognised\"\nprint(result[\"confidence\"])  # → 0.72\nprint(result[\"latency_ms\"])  # → 118\n```\n\nThree lines. No prompt, no API key, no output parsing. The embedding model (BGE-large) runs locally via ONNX, and the classifier is logistic regression coefficients trained on the standard banking77 training set. The whole thing runs on CPU\n\n```\nclient = openai.OpenAI()\n\nlabels = [\"activate_my_card\",\n  \"age_limit\", ...]  # 77 labels\n\nresponse = client.chat\n  .completions.create(\n    model=\"gpt-4o-mini\",\n    messages=[{\n      \"role\": \"user\",\n      \"content\": f\"Classify: ...\"\n    }]\n  )\n\nlabel = response.choices[0]\n  .message.content.strip()\n# hope it's valid\npython\nfrom jeffy import Engine\n\nengine = Engine()\n\nresult = engine.predict(\n  \"banking77\",\n  \"I was charged twice\"\n)\n\nlabel = result[\"label\"]\nscore = result[\"confidence\"]\n# always structured JSON\n```\n\nI ran this benchmark on a MacBook Air M2 with 16GB RAM. All LLMs via Ollama at temperature 0. 100 held-out test samples\n\n| Model | Type | Accuracy | Latency | Cost/1K | \n|---|---|---|---|---|\n| Jeffy (BGE + LR) | Embeddings | 94.3% | 118ms | $0 | \n| phi4 (14B) | LLM | 68% | 15.4s | ~$0.30 | \n| gemma4 (9.6B) | LLM | 68% | 696ms | ~$0.15 | \n| gemma3 (4B) | LLM | 64% | 956ms | ~$0.08 | \n| llama3.1 (8B) | LLM | 62% | 8.9s | ~$0.12 | \n| phi4-mini (3.8B) | LLM | 52% | 181ms | ~$0.05 | \n| llama3.2 (3B) | LLM | 46% | 143ms | ~$0.04 | \n| mistral (7B) | LLM | 39% | 17s | ~$0.12 | \n| phi (2.7B) | LLM | 30% | 231ms | ~$0.04 | \n| qwen2.5 (0.5B) | LLM | 15% | 55ms | ~$0.01 | \n\nLLMs are generalists. They weren't trained specifically on banking intent labels. When you give them 77 options like \"card_not_working\", \"contactless_not_working\", and \"card_payment_not_recognised\", they have to figure out the difference from the label names alone\n\nA trained classifier saw labeled examples during training. It learned the embedding-space boundaries between \"lost my card\" and \"card won't tap\" and \"charged twice.\" It's not guessing from label names. It knows\n\nThis matters more as class count grows. At 2 classes (positive/negative sentiment), LLMs do fine. By 77 classes, they collapse\n\n  Hosted LLMs are better than local ones at this, sure. But the fundamental problem stays: you're parsing generated text, paying per token, sending data to an external API, and dealing with seconds of latency per call. A recent paper ([\"The Embedder's Dilemma\", COLM 2026](https://arxiv.org/abs/2608.12875)) benchmarked 10 LLMs against 26 embedding models across 37 tasks and found embedding models beat LLMs on classification specifically, with LLMs costing up to 1,431x more\n\nIf you don't have a pretrained classifier for your task, training one takes seconds:\n\n``` python\n# Train a custom classifier\nfrom jeffy import Engine\n\nengine = Engine()\n\ntexts = [\"reset my password\", \"can't log in\", \"upgrade my plan\", \"cancel subscription\", ...]\nlabels = [\"account_access\", \"account_access\", \"billing\", \"billing\", ...]\n\nengine.train(\"my_support_task\", texts, labels)\n\nresult = engine.predict(\"my_support_task\", \"I forgot my login\")\n# → {\"label\": \"account_access\", \"confidence\": 0.91}\n```\n\nThe logistic regression head retrains in seconds on CPU. No GPU, no fine-tuning loop, no hyperparameter search. You can retrain it as new labeled data comes in\n\n```\npip install jeffy-classify\njeffy-serve\n```\n\n  Open `localhost:8400` and you have 68 pretrained classifiers running locally. Or try them online at [playground.jeffyclassify.com](https://playground.jeffyclassify.com)", "url": "https://wpnews.pro/news/your-llm-is-probably-overkill-for-text-classification", "canonical_source": "https://jeffyclassify.com/blog/jeffy-vs-llm-classification/", "published_at": "2026-10-21 03:15:58+00:00", "updated_at": "2026-10-21 03:18:10.300905+00:00", "lang": "en", "topics": ["machine-learning", "natural-language-processing", "ai-tools", "large-language-models"], "entities": ["Banking77", "BGE-large", "jeffy-classify", "Ollama", "phi4", "gemma4", "llama3.1", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-llm-is-probably-overkill-for-text-classification", "markdown": "https://wpnews.pro/news/your-llm-is-probably-overkill-for-text-classification.md", "text": "https://wpnews.pro/news/your-llm-is-probably-overkill-for-text-classification.txt", "jsonld": "https://wpnews.pro/news/your-llm-is-probably-overkill-for-text-classification.jsonld"}}