{"slug": "banking77-94-25-accuracy-with-a-642-kb-logistic-classifier-embeddings-sklearn", "title": "Banking77: 94.25% accuracy with a 642 KB logistic classifier (embeddings + sklearn)", "summary": "A developer published a reproducible pipeline that reaches 94.25% accuracy on the 77-class Banking77 intent-detection benchmark using frozen sentence-transformer embeddings and a scikit-learn logistic regression classifier. The classifier plus scaler weighs roughly 642 KB and trains in about 3 seconds on a MacBook Pro M3 CPU, with the smaller all-MiniLM-L6-v2 encoder hitting 93.0% in about 2 seconds. The writeup notes that strong regularization (C=0.01) and the newton-cg solver outperform default settings on high-dimensional embeddings.", "body_md": "|  | #!/usr/bin/env python3 | \n|  | \"\"\" | \n|  | Banking77: frozen embeddings + logistic regression for 77-class intent detection. | \n|  |  | \n|  | Uses the Banking77 dataset (3,080 test queries across 77 banking intents) | \n|  | to show what a simple pipeline can do: convert text to vectors with a | \n|  | pre-trained embedding model, then train an sklearn LogisticRegression on top. | \n|  | The embedding model is used as-is — no fine-tuning, no GPU needed. | \n|  |  | \n|  | The classifier + scaler weigh about 642 KB. The embedding model | \n|  | (bge-large-en-v1.5, ~1.2 GB, or all-MiniLM-L6-v2, ~91 MB) is needed | \n|  | to convert each new text into a vector before classification. | \n|  |  | \n|  | Results on a MacBook Pro M3 (CPU only): | \n|  | bge-large-en-v1.5 (1024-dim):  94.25% accuracy, ~3s classifier training | \n|  | all-MiniLM-L6-v2  (384-dim):   93.0%  accuracy, ~2s classifier training | \n|  |  | \n|  | Install: | \n|  | pip install datasets sentence-transformers scikit-learn | \n|  |  | \n|  | Run (reproduces the 93% title result with the smaller encoder): | \n|  | python banking77_gist.py | \n|  |  | \n|  | Run with the larger encoder (94.25%): | \n|  | python banking77_gist.py --model BAAI/bge-large-en-v1.5 | \n|  | \"\"\" | \n|  |  | \n|  | import argparse | \n|  | import io | \n|  | import time | \n|  |  | \n|  | import joblib | \n|  | import numpy as np | \n|  | from datasets import load_dataset | \n|  | from sentence_transformers import SentenceTransformer | \n|  | from sklearn.linear_model import LogisticRegression | \n|  | from sklearn.metrics import accuracy_score, precision_recall_fscore_support | \n|  | from sklearn.preprocessing import StandardScaler | \n|  |  | \n|  |  | \n|  | def main(): | \n|  | parser = argparse.ArgumentParser(description=\"Banking77 embeddings + logistic regression\") | \n|  | parser.add_argument( | \n|  | \"--model\", default=\"all-MiniLM-L6-v2\", | \n|  | help=\"Sentence-transformer model (default: all-MiniLM-L6-v2, ~91 MB). \" | \n|  | \"Try BAAI/bge-large-en-v1.5 (~1.2 GB) for 94.25%%.\", | \n|  | ) | \n|  | args = parser.parse_args() | \n|  |  | \n|  | # 1. Load dataset | \n|  | print(\"Loading Banking77...\") | \n|  | ds = load_dataset(\"legacy-datasets/banking77\") | \n|  | train, test = ds[\"train\"], ds[\"test\"] | \n|  | label_names = train.features[\"label\"].names | \n|  | y_train = np.array(train[\"label\"]) | \n|  | y_test = np.array(test[\"label\"]) | \n|  | print(f\"  Train: {len(y_train)}, Test: {len(y_test)}, Classes: {len(label_names)}\") | \n|  |  | \n|  | # 2. Embed with frozen sentence-transformer | \n|  | print(f\"Embedding with {args.model}...\") | \n|  | model = SentenceTransformer(args.model) | \n|  | t0 = time.time() | \n|  | X_train = model.encode(train[\"text\"], batch_size=256, show_progress_bar=True) | \n|  | X_test = model.encode(test[\"text\"], batch_size=256, show_progress_bar=True) | \n|  | embed_time = time.time() - t0 | \n|  | print(f\"  Embedding dim: {X_train.shape[1]}\") | \n|  | print(f\"  Embedding time: {embed_time:.1f}s\") | \n|  |  | \n|  | # 3. Scale features | \n|  | scaler = StandardScaler() | \n|  | X_train_s = scaler.fit_transform(X_train) | \n|  | X_test_s = scaler.transform(X_test) | \n|  |  | \n|  | # 4. Train logistic regression | \n|  | #    C=0.01 (strong regularization) works better than the default C=1.0 | \n|  | #    with high-dimensional embeddings. newton-cg is slightly better than lbfgs. | \n|  | print(\"Training classifier...\") | \n|  | t0 = time.time() | \n|  | clf = LogisticRegression( | \n|  | C=0.01, | \n|  | max_iter=3000, | \n|  | solver=\"newton-cg\", | \n|  | class_weight=\"balanced\", | \n|  | random_state=42, | \n|  | n_jobs=-1, | \n|  | ) | \n|  | clf.fit(X_train_s, y_train) | \n|  | train_time = time.time() - t0 | \n|  | print(f\"  Training time: {train_time:.2f}s\") | \n|  |  | \n|  | # 5. Evaluate | \n|  | y_pred = clf.predict(X_test_s) | \n|  | accuracy = accuracy_score(y_test, y_pred) | \n|  |  | \n|  | # Classifier + scaler size (the embedding encoder is separate) | \n|  | buf = io.BytesIO() | \n|  | joblib.dump((clf, scaler), buf) | \n|  | classifier_size_kb = buf.tell() / 1024 | \n|  |  | \n|  | print(f\"\\n{'='*50}\") | \n|  | print(f\"RESULTS\") | \n|  | print(f\"{'='*50}\") | \n|  | print(f\"  Encoder:          {args.model}\") | \n|  | print(f\"  Accuracy:         {accuracy:.4f} ({accuracy*100:.2f}%)\") | \n|  | print(f\"  Correct:          {int(accuracy * len(y_test))}/{len(y_test)}\") | \n|  | print(f\"  Classes:          {len(label_names)}\") | \n|  | print(f\"  Classifier+scaler: {classifier_size_kb:.0f} KB\") | \n|  | print(f\"  Classifier train: {train_time:.2f}s\") | \n|  | print(f\"  Embedding time:   {embed_time:.1f}s\") | \n|  |  | \n|  | # 6. Per-class accuracy (worst and best) | \n|  | p, r, f, s = precision_recall_fscore_support(y_test, y_pred, average=None) | \n|  | print(f\"\\n  Hardest classes:\") | \n|  | for i in np.argsort(f)[:5]: | \n|  | print(f\"    {label_names[i]:<45s} F1={f[i]:.3f}\") | \n|  | print(f\"  Easiest classes:\") | \n|  | for i in np.argsort(f)[-5:][::-1]: | \n|  | print(f\"    {label_names[i]:<45s} F1={f[i]:.3f}\") | \n|  |  | \n|  | # 7. Quick inference demo | \n|  | print(f\"\\n  Example predictions:\") | \n|  | examples = [ | \n|  | \"I want to know where my package is\", | \n|  | \"My card was declined at the store\", | \n|  | \"How do I change my PIN number?\", | \n|  | \"I need to cancel my subscription\", | \n|  | \"What's the exchange rate for euros?\", | \n|  | ] | \n|  | X_ex = model.encode(examples) | \n|  | X_ex_s = scaler.transform(X_ex) | \n|  | preds = clf.predict(X_ex_s) | \n|  | probas = clf.predict_proba(X_ex_s) | \n|  | for text, pred_idx, proba in zip(examples, preds, probas): | \n|  | conf = proba[pred_idx] | \n|  | print(f'    \"{text}\"') | \n|  | print(f\"      -> {label_names[pred_idx]} ({conf:.0%})\") | \n|  |  | \n|  |  | \n|  | if __name__ == \"__main__\": | \n|  | main() |", "url": "https://wpnews.pro/news/banking77-94-25-accuracy-with-a-642-kb-logistic-classifier-embeddings-sklearn", "canonical_source": "https://gist.github.com/nicobrenner/056a5aaff5d0119c0032ecdad5029557", "published_at": "2026-09-24 16:23:06+00:00", "updated_at": "2026-10-16 17:51:29.840995+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "ai-research", "ai-tools"], "entities": ["Banking77", "scikit-learn", "sentence-transformers", "BAAI/bge-large-en-v1.5", "all-MiniLM-L6-v2", "MacBook Pro M3"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/banking77-94-25-accuracy-with-a-642-kb-logistic-classifier-embeddings-sklearn", "markdown": "https://wpnews.pro/news/banking77-94-25-accuracy-with-a-642-kb-logistic-classifier-embeddings-sklearn.md", "text": "https://wpnews.pro/news/banking77-94-25-accuracy-with-a-642-kb-logistic-classifier-embeddings-sklearn.txt", "jsonld": "https://wpnews.pro/news/banking77-94-25-accuracy-with-a-642-kb-logistic-classifier-embeddings-sklearn.jsonld"}}