Your LLM is probably overkill for text classification

I keep seeing the same pattern. Someone needs to classify text (route support tickets, tag documents, detect intent) so they prompt an LLM. It works in the demo. A few months later the complaints are about latency, cost, and malformed JSON breaking things downstream

There's a simpler approach that gets better results on classification: freeze a sentence embedding model and train a logistic regression head on top. That's it. No fine-tuning, no GPU, no prompt engineering

I'll show both approaches on the same task so you can see the difference

Banking77 is a customer intent dataset with 77 classes. Things like "lost_or_stolen_card", "card_payment_not_recognised", "contactless_not_working". Many labels are semantically close, which is what makes it a real test

This is the kind of problem that shows up everywhere: support ticket routing, email classification, voice assistant intent detection. Lots of categories, subtle differences between them

Here's how you'd classify text with an LLM. Using OpenAI as the example since that's what most people reach for:

import openai
import json

client = openai.OpenAI()

labels = [
    "activate_my_card", "age_limit", "apple_pay_or_google_pay",
    "atm_support", "automatic_top_up", "balance_not_updated_after_bank_transfer",
    "balance_not_updated_after_cheque_or_cash_deposit", "beneficiary_not_allowed",
    "cancel_transfer", "card_about_to_expire", "card_acceptance",
    "virtual_card_not_working", "visa_or_mastercard",
    "why_verify_identity", "wrong_amount_of_cash_received",
    "wrong_exchange_rate_for_cash_withdrawal"
]

def classify_with_llm(text):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[{
            "role": "user",
            "content": f"""Classify this customer message into exactly
one of these categories: {json.dumps(labels)}

Message: {text}

Reply with ONLY the category name, nothing else."""
        }]
    )
    return response.choices[0].message.content.strip()

result = classify_with_llm("I was charged twice for the same transaction")

Problems with this:

Here's the same task with Jeffy:

pip install jeffy-classify
python
from jeffy import Engine

engine = Engine()
result = engine.predict("banking77", "I was charged twice for the same transaction")

print(result["label"])       # → "card_payment_not_recognised"
print(result["confidence"])  # → 0.72
print(result["latency_ms"])  # → 118

Three lines. No prompt, no API key, no output parsing. The embedding model (BGE-large) runs locally via ONNX, and the classifier is logistic regression coefficients trained on the standard banking77 training set. The whole thing runs on CPU

client = openai.OpenAI()

labels = ["activate_my_card",
  "age_limit", ...]  # 77 labels

response = client.chat
  .completions.create(
    model="gpt-4o-mini",
    messages=[{
      "role": "user",
      "content": f"Classify: ..."
    }]
  )

label = response.choices[0]
  .message.content.strip()
python
from jeffy import Engine

engine = Engine()

result = engine.predict(
  "banking77",
  "I was charged twice"
)

label = result["label"]
score = result["confidence"]

I ran this benchmark on a MacBook Air M2 with 16GB RAM. All LLMs via Ollama at temperature 0. 100 held-out test samples

LLMs are generalists. They weren't trained specifically on banking intent labels. When you give them 77 options like "card_not_working", "contactless_not_working", and "card_payment_not_recognised", they have to figure out the difference from the label names alone

A trained classifier saw labeled examples during training. It learned the embedding-space boundaries between "lost my card" and "card won't tap" and "charged twice." It's not guessing from label names. It knows

This matters more as class count grows. At 2 classes (positive/negative sentiment), LLMs do fine. By 77 classes, they collapse

Hosted LLMs are better than local ones at this, sure. But the fundamental problem stays: you're parsing generated text, paying per token, sending data to an external API, and dealing with seconds of latency per call. A recent paper ("The Embedder's Dilemma", COLM 2026) benchmarked 10 LLMs against 26 embedding models across 37 tasks and found embedding models beat LLMs on classification specifically, with LLMs costing up to 1,431x more

If you don't have a pretrained classifier for your task, training one takes seconds:

from jeffy import Engine

engine = Engine()

texts = ["reset my password", "can't log in", "upgrade my plan", "cancel subscription", ...]
labels = ["account_access", "account_access", "billing", "billing", ...]

engine.train("my_support_task", texts, labels)

result = engine.predict("my_support_task", "I forgot my login")

The logistic regression head retrains in seconds on CPU. No GPU, no fine-tuning loop, no hyperparameter search. You can retrain it as new labeled data comes in

pip install jeffy-classify
jeffy-serve

Open localhost:8400 and you have 68 pretrained classifiers running locally. Or try them online at playground.jeffyclassify.com