Fine-Tuning vs. Frozen Embeddings: When to Adapt the LLM vs. the Sklearn Head

A practical comparison of LoRA fine-tuning versus training a scikit-learn classifier on frozen embeddings found that the two strategies differ in training time and classification accuracy on the ag_news dataset, using the Qwen/Qwen2.5-1.5B model with 800 training and 200 test examples. The walkthrough implements QLoRA fine-tuning via the PEFT library with a Hugging Face classification head and contrasts it with a lightweight frozen-embedding approach for compute-constrained applications. The comparison is aimed at helping practitioners decide when adapting the LLM itself is necessary versus when a downstream classifier suffices.

In this article, you will learn how to compare two strategies for adapting a large language model to a specific domain: fine-tuning with LoRA versus training a scikit-learn classifier on frozen embeddings. Topics we will cover include: - How frozen embeddings work as a lightweight alternative to full model fine-tuning, and when this approach is appropriate. - How to implement QLoRA fine-tuning using the PEFT library with a Hugging Face model and a classification head. - How the two strategies compare in terms of training time and classification accuracy on a real dataset. Introduction When adapting an LLM a large language model to a specific domain, the classical approach picked by the machine learning community boils down to fine-tuning the model on a domain-specific dataset. But a question arises: is this always necessary? Or could alternative strategies like using frozen embeddings combined with lightweight downstream classifiers on top of the model be a wiser approach to adopt, especially when building cost-friendly, compute-constrained applications? This article presents a practical comparison of fine-tuning a local LLM with LoRA Low-Rank Adaptation, an algorithm for efficient weight updating against training only a downstream scikit-learn classifier on frozen embeddings. We cover, analyze, and discuss when each strategy is preferred based on cost and performance. To make the most of the practical example below, we recommend a basic level of familiarity with LLMs, key processes like model fine-tuning, and associated libraries like Hugging Face’s transformers . Initial Setup If you are running this practical comparison on Google Colab, first and foremost, make sure you set the running environment to GPU T4 , as the code and methods used in it are optimized for GPU use. You will only need to install the following libraries: pip install -q -U bitsandbytes peft accelerate evaluate 1 pip install -q -U bitsandbytes peft accelerate evaluate Otherwise, if you are in a different Python environment, make sure to have scikit-learn , transformers , and datasets also installed before proceeding. We start by making some of the imports needed and loading a canonical dataset from the Hugging Face repository: ag news , containing categorized news articles. python import torch import numpy as np import time from datasets import load dataset from transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig To use Llama-3 instead, change this to "meta-llama/Meta-Llama-3-8B" MODEL ID = "Qwen/Qwen2.5-1.5B" print "1. Loading dataset..." dataset = load dataset "fancyzhx/ag news" A tiny subset is used so that the comparison can be run in minutes, not hours, on GPU train subset = dataset "train" .shuffle seed=42 .select range 800 test subset = dataset "test" .shuffle seed=42 .select range 200 1234567891011121314 import torchimport numpy as npimport timefrom datasets import load datasetfrom transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig To use Llama-3 instead, change this to "meta-llama/Meta-Llama-3-8B" MODEL ID = "Qwen/Qwen2.5-1.5B" print "1. Loading dataset..." dataset = load dataset "fancyzhx/ag news" A tiny subset is used so that the comparison can be run in minutes, not hours, on GPUtrain subset = dataset "train" .shuffle seed=42 .select range 800 test subset = dataset "test" .shuffle seed=42 .select range 200 Next, we load the model instance and its associated tokenizer, ensuring it is configured with a classification head compatible with the specific problem dataset at hand, in which news articles are classified into four possible classes: print "2. Configuring Tokenizer..." tokenizer = AutoTokenizer.from pretrained MODEL ID tokenizer.pad token = tokenizer.eos token Necessary for decoder-only models print "3. Loading Base Model in 4-bit..." bnb config = BitsAndBytesConfig load in 4bit=True, bnb 4bit compute dtype=torch.bfloat16, bnb 4bit quant type="nf4" Loading the model with a classification head model = AutoModelForSequenceClassification.from pretrained MODEL ID, num labels=4, quantization config=bnb config, device map="auto" model.config.pad token id = tokenizer.pad token id 12345678910111213141516171819 print "2. Configuring Tokenizer..." tokenizer = AutoTokenizer.from pretrained MODEL ID tokenizer.pad token = tokenizer.eos token Necessary for decoder-only models print "3. Loading Base Model in 4-bit..." bnb config = BitsAndBytesConfig load in 4bit=True, bnb 4bit compute dtype=torch.bfloat16, bnb 4bit quant type="nf4" Loading the model with a classification headmodel = AutoModelForSequenceClassification.from pretrained MODEL ID, num labels=4, quantization config=bnb config, device map="auto" model.config.pad token id = tokenizer.pad token id Strategy A: Frozen Embeddings & Scikit-learn We start by trying out the alternative to fine-tuning our model. It works as follows: instead of retraining the LLM, we make it act strictly as a feature extractor. We pass the text through this “frozen model” one in which weights cannot change and extract the last hidden state the embedding vector for the final token. These high-dimensional embeddings are then fed to an auxiliary model — in other words, they are used to train a traditional lightweight classifier on top of the frozen LLM. The first step is defining a function with the internal logic to obtain the frozen embeddings from input texts: python from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import accuracy score def extract embeddings texts, batch size=8 : model.eval embeddings = for i in range 0, len texts , batch size : batch = texts i:i+batch size inputs = tokenizer batch, return tensors="pt", padding=True, truncation=True, max length=128 .to model.device with torch.no grad : output hidden states=True forces the model to return the raw vectors outputs = model inputs, output hidden states=True last hidden state = outputs.hidden states -1 Mean pooling: averaging the vectors across the sequence length, ignoring padding tokens mask = inputs "attention mask" .unsqueeze -1 .float sum embeds = torch.sum last hidden state mask, dim=1 mean pooled = sum embeds / torch.clamp mask.sum dim=1 , min=1e-9 embeddings.append mean pooled.cpu .numpy .astype np.float32 return np.concatenate embeddings, axis=0 123456789101112131415161718192021222324 from sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import accuracy score def extract embeddings texts, batch size=8 : model.eval embeddings = for i in range 0, len texts , batch size : batch = texts i:i+batch size inputs = tokenizer batch, return tensors="pt", padding=True, truncation=True, max length=128 .to model.device with torch.no grad : output hidden states=True forces the model to return the raw vectors outputs = model inputs, output hidden states=True last hidden state = outputs.hidden states -1 Mean pooling: averaging the vectors across the sequence length, ignoring padding tokens mask = inputs "attention mask" .unsqueeze -1 .float sum embeds = torch.sum last hidden state mask, dim=1 mean pooled = sum embeds / torch.clamp mask.sum dim=1 , min=1e-9 embeddings.append mean pooled.cpu .numpy .astype np.float32 return np.concatenate embeddings, axis=0 Next, we call the function passing our dataset sample — already split into training and test subsets — and obtain the frozen embeddings that will be used to train and test a random forest classifier: print "Extracting embeddings..." start time = time.time X train = extract embeddings train subset "text" X test = extract embeddings test subset "text" y train = train subset "label" y test = test subset "label" embed time = time.time - start time print "Training Scikit-Learn Random Forest..." clf = RandomForestClassifier n estimators=100, random state=42 start train time = time.time clf.fit X train, y train sklearn train time = time.time - start train time preds = clf.predict X test sklearn acc = accuracy score y test, preds print f"\n--- Strategy A Results ---" print f"Extraction Time: {embed time:.1f}s" print f"Classifier Train Time: {sklearn train time:.3f}s" print f"Accuracy: {sklearn acc 100:.1f}%" 123456789101112131415161718192021 print "Extracting embeddings..." start time = time.time X train = extract embeddings train subset "text" X test = extract embeddings test subset "text" y train = train subset "label" y test = test subset "label" embed time = time.time - start time print "Training Scikit-Learn Random Forest..." clf = RandomForestClassifier n estimators=100, random state=42 start train time = time.time clf.fit X train, y train sklearn train time = time.time - start train time preds = clf.predict X test sklearn acc = accuracy score y test, preds print f"\n--- Strategy A Results ---" print f"Extraction Time: {embed time:.1f}s" print f"Classifier Train Time: {sklearn train time:.3f}s" print f"Accuracy: {sklearn acc 100:.1f}%" Results: --- Strategy A Results --- Extraction Time: 96.5s Classifier Train Time: 2.866s Accuracy: 82.0% 1234 --- Strategy A Results ---Extraction Time: 96.5sClassifier Train Time: 2.866sAccuracy: 82.0% While it took nearly 100 seconds to complete, the vast majority of that time went to the feature extraction process — the classifier training itself was almost instant. Let’s see whether the 82% test classification accuracy yielded by this frozen embedding-driven classifier can be beaten by a fine-tuned LLM. Strategy B: LoRA Fine-Tuning QLoRA This strategy uses Low-Rank Adaptation LoRA : an optimized fine-tuning algorithm that freezes the massive pre-trained weights of the model and injects trainable “adapter” weight matrices into the transformer layers. The complexity of the process boils down to training these small adapter matrices, which typically account for just 1-2% of the total model size, while maintaining 4-bit precision to save memory. The first two steps are attaching the adapters to the base model and preparing the data for fine-tuning: python from peft import LoraConfig, get peft model, prepare model for kbit training from transformers import TrainingArguments, Trainer import evaluate print "1. Attaching LoRA Adapters to Base Model..." model = prepare model for kbit training model peft config = LoraConfig task type="SEQ CLS", r=16, lora alpha=32, lora dropout=0.05, target modules= "q proj", "v proj" Target attention layers lora model = get peft model model, peft config lora model.print trainable parameters print "\n2. Preparing Data for Trainer..." def tokenize fn examples : return tokenizer examples "text" , padding="max length", truncation=True, max length=128 tokenized train = train subset.map tokenize fn, batched=True tokenized test = test subset.map tokenize fn, batched=True accuracy metric = evaluate.load "accuracy" def compute metrics eval pred : predictions, labels = eval pred predictions = np.argmax predictions, axis=1 return accuracy metric.compute predictions=predictions, references=labels 12345678910111213141516171819202122232425262728 from peft import LoraConfig, get peft model, prepare model for kbit trainingfrom transformers import TrainingArguments, Trainerimport evaluate print "1. Attaching LoRA Adapters to Base Model..." model = prepare model for kbit training model peft config = LoraConfig task type="SEQ CLS", r=16, lora alpha=32, lora dropout=0.05, target modules= "q proj", "v proj" Target attention layers lora model = get peft model model, peft config lora model.print trainable parameters print "\n2. Preparing Data for Trainer..." def tokenize fn examples : return tokenizer examples "text" , padding="max length", truncation=True, max length=128 tokenized train = train subset.map tokenize fn, batched=True tokenized test = test subset.map tokenize fn, batched=True accuracy metric = evaluate.load "accuracy" def compute metrics eval pred : predictions, labels = eval pred predictions = np.argmax predictions, axis=1 return accuracy metric.compute predictions=predictions, references=labels Note that the result of integrating the adapter matrices into our base LLM is stored in a new model variable named lora model . We then conduct the core fine-tuning process configured as LoRA — it requires TrainingArguments and Trainer objects: print "\n3. Starting QLoRA Fine-Tuning..." training args = TrainingArguments output dir="./results", per device train batch size=8, per device eval batch size=8, learning rate=2e-4, num train epochs=3, eval strategy="epoch", logging strategy="epoch", optim="paged adamw 8bit", Saves VRAM by offloading optimizer states fp16=True, report to="none" trainer = Trainer model=lora model, args=training args, train dataset=tokenized train, eval dataset=tokenized test, compute metrics=compute metrics, start time = time.time trainer.train lora train time = time.time - start time print "\n4. Evaluating QLoRA Model..." eval results = trainer.evaluate lora acc = eval results "eval accuracy" print f"\n--- Strategy B Results ---" print f"LoRA Train Time: {lora train time:.1f}s" print f"Accuracy: {lora acc 100:.1f}%" 123456789101112131415161718192021222324252627282930313233 print "\n3. Starting QLoRA Fine-Tuning..." training args = TrainingArguments output dir="./results", per device train batch size=8, per device eval batch size=8, learning rate=2e-4, num train epochs=3, eval strategy="epoch", logging strategy="epoch", optim="paged adamw 8bit", Saves VRAM by offloading optimizer states fp16=True, report to="none" trainer = Trainer model=lora model, args=training args, train dataset=tokenized train, eval dataset=tokenized test, compute metrics=compute metrics, start time = time.time trainer.train lora train time = time.time - start time print "\n4. Evaluating QLoRA Model..." eval results = trainer.evaluate lora acc = eval results "eval accuracy" print f"\n--- Strategy B Results ---" print f"LoRA Train Time: {lora train time:.1f}s" print f"Accuracy: {lora acc 100:.1f}%" Results you will see more detailed results per training epoch if you execute the code : --- Strategy B Results --- LoRA Train Time: 300.3s Accuracy: 87.0% 123 --- Strategy B Results ---LoRA Train Time: 300.3sAccuracy: 87.0% Classification performance does improve when we adapt our model to the specific problem the traditional way: by fine-tuning it. That improvement comes at a cost, however — the entire process took three times as long, with all of that overhead going into the training process itself. For larger datasets, this difference in time may become substantially more pronounced. Verdict and Conclusion Feeding frozen embeddings to an auxiliary scikit-learn model is the recommended approach when seeking rapid training and minimal compute overhead while avoiding backbone overfitting. QLoRA fine-tuning, by contrast, is the better choice when closing the accuracy gap is critical to your domain, or when the model needs to learn deeply embedded domain-specific nuances. This calculus changes for purely generative tasks like text and code generation, where model adaptation — such as fine-tuning — is typically necessary for specialized behaviors. That said, the frozen embedding strategy can sometimes be a viable alternative for generative workflows in free-tier deployments where maximum performance is slightly negotiable.