Banking77: 94.25% accuracy with a 642 KB logistic classifier (embeddings + sklearn)

A developer published a reproducible pipeline that reaches 94.25% accuracy on the 77-class Banking77 intent-detection benchmark using frozen sentence-transformer embeddings and a scikit-learn logistic regression classifier. The classifier plus scaler weighs roughly 642 KB and trains in about 3 seconds on a MacBook Pro M3 CPU, with the smaller all-MiniLM-L6-v2 encoder hitting 93.0% in about 2 seconds. The writeup notes that strong regularization (C=0.01) and the newton-cg solver outperform default settings on high-dimensional embeddings.

| | /usr/bin/env python3 | | | """ | | | Banking77: frozen embeddings + logistic regression for 77-class intent detection. | | | | | | Uses the Banking77 dataset 3,080 test queries across 77 banking intents | | | to show what a simple pipeline can do: convert text to vectors with a | | | pre-trained embedding model, then train an sklearn LogisticRegression on top. | | | The embedding model is used as-is — no fine-tuning, no GPU needed. | | | | | | The classifier + scaler weigh about 642 KB. The embedding model | | | bge-large-en-v1.5, ~1.2 GB, or all-MiniLM-L6-v2, ~91 MB is needed | | | to convert each new text into a vector before classification. | | | | | | Results on a MacBook Pro M3 CPU only : | | | bge-large-en-v1.5 1024-dim : 94.25% accuracy, ~3s classifier training | | | all-MiniLM-L6-v2 384-dim : 93.0% accuracy, ~2s classifier training | | | | | | Install: | | | pip install datasets sentence-transformers scikit-learn | | | | | | Run reproduces the 93% title result with the smaller encoder : | | | python banking77 gist.py | | | | | | Run with the larger encoder 94.25% : | | | python banking77 gist.py --model BAAI/bge-large-en-v1.5 | | | """ | | | | | | import argparse | | | import io | | | import time | | | | | | import joblib | | | import numpy as np | | | from datasets import load dataset | | | from sentence transformers import SentenceTransformer | | | from sklearn.linear model import LogisticRegression | | | from sklearn.metrics import accuracy score, precision recall fscore support | | | from sklearn.preprocessing import StandardScaler | | | | | | | | | def main : | | | parser = argparse.ArgumentParser description="Banking77 embeddings + logistic regression" | | | parser.add argument | | | "--model", default="all-MiniLM-L6-v2", | | | help="Sentence-transformer model default: all-MiniLM-L6-v2, ~91 MB . " | | | "Try BAAI/bge-large-en-v1.5 ~1.2 GB for 94.25%%.", | | | | | | args = parser.parse args | | | | | | 1. Load dataset | | | print "Loading Banking77..." | | | ds = load dataset "legacy-datasets/banking77" | | | train, test = ds "train" , ds "test" | | | label names = train.features "label" .names | | | y train = np.array train "label" | | | y test = np.array test "label" | | | print f" Train: {len y train }, Test: {len y test }, Classes: {len label names }" | | | | | | 2. Embed with frozen sentence-transformer | | | print f"Embedding with {args.model}..." | | | model = SentenceTransformer args.model | | | t0 = time.time | | | X train = model.encode train "text" , batch size=256, show progress bar=True | | | X test = model.encode test "text" , batch size=256, show progress bar=True | | | embed time = time.time - t0 | | | print f" Embedding dim: {X train.shape 1 }" | | | print f" Embedding time: {embed time:.1f}s" | | | | | | 3. Scale features | | | scaler = StandardScaler | | | X train s = scaler.fit transform X train | | | X test s = scaler.transform X test | | | | | | 4. Train logistic regression | | | C=0.01 strong regularization works better than the default C=1.0 | | | with high-dimensional embeddings. newton-cg is slightly better than lbfgs. | | | print "Training classifier..." | | | t0 = time.time | | | clf = LogisticRegression | | | C=0.01, | | | max iter=3000, | | | solver="newton-cg", | | | class weight="balanced", | | | random state=42, | | | n jobs=-1, | | | | | | clf.fit X train s, y train | | | train time = time.time - t0 | | | print f" Training time: {train time:.2f}s" | | | | | | 5. Evaluate | | | y pred = clf.predict X test s | | | accuracy = accuracy score y test, y pred | | | | | | Classifier + scaler size the embedding encoder is separate | | | buf = io.BytesIO | | | joblib.dump clf, scaler , buf | | | classifier size kb = buf.tell / 1024 | | | | | | print f"\n{'=' 50}" | | | print f"RESULTS" | | | print f"{'=' 50}" | | | print f" Encoder: {args.model}" | | | print f" Accuracy: {accuracy:.4f} {accuracy 100:.2f}% " | | | print f" Correct: {int accuracy len y test }/{len y test }" | | | print f" Classes: {len label names }" | | | print f" Classifier+scaler: {classifier size kb:.0f} KB" | | | print f" Classifier train: {train time:.2f}s" | | | print f" Embedding time: {embed time:.1f}s" | | | | | | 6. Per-class accuracy worst and best | | | p, r, f, s = precision recall fscore support y test, y pred, average=None | | | print f"\n Hardest classes:" | | | for i in np.argsort f :5 : | | | print f" {label names i :<45s} F1={f i :.3f}" | | | print f" Easiest classes:" | | | for i in np.argsort f -5: ::-1 : | | | print f" {label names i :<45s} F1={f i :.3f}" | | | | | | 7. Quick inference demo | | | print f"\n Example predictions:" | | | examples = | | | "I want to know where my package is", | | | "My card was declined at the store", | | | "How do I change my PIN number?", | | | "I need to cancel my subscription", | | | "What's the exchange rate for euros?", | | | | | | X ex = model.encode examples | | | X ex s = scaler.transform X ex | | | preds = clf.predict X ex s | | | probas = clf.predict proba X ex s | | | for text, pred idx, proba in zip examples, preds, probas : | | | conf = proba pred idx | | | print f' "{text}"' | | | print f" - {label names pred idx } {conf:.0%} " | | | | | | | | | if name == " main ": | | | main |