{"slug": "fine-tuning-vs-frozen-embeddings-when-to-adapt-the-llm-vs-the-sklearn-head", "title": "Fine-Tuning vs. Frozen Embeddings: When to Adapt the LLM vs. the Sklearn Head", "summary": "A practical comparison of LoRA fine-tuning versus training a scikit-learn classifier on frozen embeddings found that the two strategies differ in training time and classification accuracy on the ag_news dataset, using the Qwen/Qwen2.5-1.5B model with 800 training and 200 test examples. The walkthrough implements QLoRA fine-tuning via the PEFT library with a Hugging Face classification head and contrasts it with a lightweight frozen-embedding approach for compute-constrained applications. The comparison is aimed at helping practitioners decide when adapting the LLM itself is necessary versus when a downstream classifier suffices.", "body_md": "In this article, you will learn how to compare two strategies for adapting a large language model to a specific domain: fine-tuning with LoRA versus training a scikit-learn classifier on frozen embeddings.\n\nTopics we will cover include:\n\n- How frozen embeddings work as a lightweight alternative to full model fine-tuning, and when this approach is appropriate.\n- How to implement QLoRA fine-tuning using the PEFT library with a Hugging Face model and a classification head.\n- How the two strategies compare in terms of training time and classification accuracy on a real dataset.\n\n## Introduction\n\nWhen adapting an LLM (a large language model) to a specific domain, the classical approach picked by the machine learning community boils down to **fine-tuning** the model on a domain-specific dataset. But a question arises: is this always necessary? Or could alternative strategies like using **frozen embeddings** combined with lightweight downstream classifiers on top of the model be a wiser approach to adopt, especially when building cost-friendly, compute-constrained applications?\n\nThis article presents a practical comparison of fine-tuning a local LLM with LoRA (Low-Rank Adaptation, an algorithm for efficient weight updating) against training only a downstream scikit-learn classifier on frozen embeddings. We cover, analyze, and discuss when each strategy is preferred based on cost and performance.\n\nTo make the most of the practical example below, we recommend a basic level of familiarity with LLMs, key processes like model fine-tuning, and associated libraries like Hugging Face’s `transformers`.\n\n## Initial Setup\n\nIf you are running this practical comparison on Google Colab, first and foremost, make sure you set the running environment to GPU (T4), as the code and methods used in it are optimized for GPU use. You will only need to install the following libraries:\n\n```\n!pip install -q -U bitsandbytes peft accelerate evaluate\n\n1\n\n!pip install -q -U bitsandbytes peft accelerate evaluate\n```\n\nOtherwise, if you are in a different Python environment, make sure to have `scikit-learn`, `transformers`, and `datasets` also installed before proceeding.\n\nWe start by making some of the imports needed and loading a canonical dataset from the Hugging Face repository: `ag_news`, containing categorized news articles.\n\n``` python\nimport torch\nimport numpy as np\nimport time\nfrom datasets import load_dataset\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig\n\n# To use Llama-3 instead, change this to \"meta-llama/Meta-Llama-3-8B\" \nMODEL_ID = \"Qwen/Qwen2.5-1.5B\" \n\nprint(\"1. Loading dataset...\")\ndataset = load_dataset(\"fancyzhx/ag_news\")\n# A tiny subset is used so that the comparison can be run in minutes, not hours, on GPU\ntrain_subset = dataset[\"train\"].shuffle(seed=42).select(range(800))\ntest_subset = dataset[\"test\"].shuffle(seed=42).select(range(200))\n\n1234567891011121314\n\nimport torchimport numpy as npimport timefrom datasets import load_datasetfrom transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig # To use Llama-3 instead, change this to \"meta-llama/Meta-Llama-3-8B\" MODEL_ID = \"Qwen/Qwen2.5-1.5B\"  print(\"1. Loading dataset...\")dataset = load_dataset(\"fancyzhx/ag_news\")# A tiny subset is used so that the comparison can be run in minutes, not hours, on GPUtrain_subset = dataset[\"train\"].shuffle(seed=42).select(range(800))test_subset = dataset[\"test\"].shuffle(seed=42).select(range(200))\n```\n\nNext, we load the model instance and its associated tokenizer, ensuring it is configured with a classification head compatible with the specific problem (dataset) at hand, in which news articles are classified into four possible classes:\n\n```\nprint(\"2. Configuring Tokenizer...\")\ntokenizer = AutoTokenizer.from_pretrained(MODEL_ID)\ntokenizer.pad_token = tokenizer.eos_token # Necessary for decoder-only models\n\nprint(\"3. Loading Base Model in 4-bit...\")\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.bfloat16,\n    bnb_4bit_quant_type=\"nf4\"\n)\n\n# Loading the model with a classification head\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    MODEL_ID,\n    num_labels=4,\n    quantization_config=bnb_config,\n    device_map=\"auto\"\n)\nmodel.config.pad_token_id = tokenizer.pad_token_id\n\n12345678910111213141516171819\n\nprint(\"2. Configuring Tokenizer...\")tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)tokenizer.pad_token = tokenizer.eos_token # Necessary for decoder-only models print(\"3. Loading Base Model in 4-bit...\")bnb_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_compute_dtype=torch.bfloat16,    bnb_4bit_quant_type=\"nf4\") # Loading the model with a classification headmodel = AutoModelForSequenceClassification.from_pretrained(    MODEL_ID,    num_labels=4,    quantization_config=bnb_config,    device_map=\"auto\")model.config.pad_token_id = tokenizer.pad_token_id\n```\n\n## Strategy A: Frozen Embeddings & Scikit-learn\n\nWe start by trying out the alternative to fine-tuning our model. It works as follows: instead of retraining the LLM, we make it act strictly as a feature extractor. We pass the text through this “frozen model” (one in which weights cannot change) and extract the last hidden state (the embedding vector) for the final token. These high-dimensional embeddings are then fed to an auxiliary model — in other words, they are used to train a traditional lightweight classifier on top of the frozen LLM.\n\nThe first step is defining a function with the internal logic to obtain the frozen embeddings from input texts:\n\n``` python\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import accuracy_score\n\ndef extract_embeddings(texts, batch_size=8):\n    model.eval()\n    embeddings = []\n    \n    for i in range(0, len(texts), batch_size):\n        batch = texts[i:i+batch_size]\n        inputs = tokenizer(batch, return_tensors=\"pt\", padding=True, truncation=True, max_length=128).to(model.device)\n        \n        with torch.no_grad():\n            # output_hidden_states=True forces the model to return the raw vectors\n            outputs = model(**inputs, output_hidden_states=True)\n            last_hidden_state = outputs.hidden_states[-1]\n            \n            # Mean pooling: averaging the vectors across the sequence length, ignoring padding tokens\n            mask = inputs[\"attention_mask\"].unsqueeze(-1).float()\n            sum_embeds = torch.sum(last_hidden_state * mask, dim=1)\n            mean_pooled = sum_embeds / torch.clamp(mask.sum(dim=1), min=1e-9)\n            \n            embeddings.append(mean_pooled.cpu().numpy().astype(np.float32))\n            \n    return np.concatenate(embeddings, axis=0)\n\n123456789101112131415161718192021222324\n\nfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import accuracy_score def extract_embeddings(texts, batch_size=8):    model.eval()    embeddings = []        for i in range(0, len(texts), batch_size):        batch = texts[i:i+batch_size]        inputs = tokenizer(batch, return_tensors=\"pt\", padding=True, truncation=True, max_length=128).to(model.device)                with torch.no_grad():            # output_hidden_states=True forces the model to return the raw vectors            outputs = model(**inputs, output_hidden_states=True)            last_hidden_state = outputs.hidden_states[-1]                        # Mean pooling: averaging the vectors across the sequence length, ignoring padding tokens            mask = inputs[\"attention_mask\"].unsqueeze(-1).float()            sum_embeds = torch.sum(last_hidden_state * mask, dim=1)            mean_pooled = sum_embeds / torch.clamp(mask.sum(dim=1), min=1e-9)                        embeddings.append(mean_pooled.cpu().numpy().astype(np.float32))                return np.concatenate(embeddings, axis=0)\n```\n\nNext, we call the function passing our dataset sample — already split into training and test subsets — and obtain the frozen embeddings that will be used to train and test a random forest classifier:\n\n```\nprint(\"Extracting embeddings...\")\nstart_time = time.time()\nX_train = extract_embeddings(train_subset[\"text\"])\nX_test = extract_embeddings(test_subset[\"text\"])\ny_train = train_subset[\"label\"]\ny_test = test_subset[\"label\"]\nembed_time = time.time() - start_time\n\nprint(\"Training Scikit-Learn Random Forest...\")\nclf = RandomForestClassifier(n_estimators=100, random_state=42)\nstart_train_time = time.time()\nclf.fit(X_train, y_train)\nsklearn_train_time = time.time() - start_train_time\n\npreds = clf.predict(X_test)\nsklearn_acc = accuracy_score(y_test, preds)\n\nprint(f\"\\n--- Strategy A Results ---\")\nprint(f\"Extraction Time: {embed_time:.1f}s\")\nprint(f\"Classifier Train Time: {sklearn_train_time:.3f}s\")\nprint(f\"Accuracy: {sklearn_acc * 100:.1f}%\")\n\n123456789101112131415161718192021\n\nprint(\"Extracting embeddings...\")start_time = time.time()X_train = extract_embeddings(train_subset[\"text\"])X_test = extract_embeddings(test_subset[\"text\"])y_train = train_subset[\"label\"]y_test = test_subset[\"label\"]embed_time = time.time() - start_time print(\"Training Scikit-Learn Random Forest...\")clf = RandomForestClassifier(n_estimators=100, random_state=42)start_train_time = time.time()clf.fit(X_train, y_train)sklearn_train_time = time.time() - start_train_time preds = clf.predict(X_test)sklearn_acc = accuracy_score(y_test, preds) print(f\"\\n--- Strategy A Results ---\")print(f\"Extraction Time: {embed_time:.1f}s\")print(f\"Classifier Train Time: {sklearn_train_time:.3f}s\")print(f\"Accuracy: {sklearn_acc * 100:.1f}%\")\n```\n\nResults:\n\n```\n--- Strategy A Results ---\nExtraction Time: 96.5s\nClassifier Train Time: 2.866s\nAccuracy: 82.0%\n\n1234\n\n--- Strategy A Results ---Extraction Time: 96.5sClassifier Train Time: 2.866sAccuracy: 82.0%\n```\n\nWhile it took nearly 100 seconds to complete, the vast majority of that time went to the feature extraction process — the classifier training itself was almost instant.\n\nLet’s see whether the 82% test classification accuracy yielded by this frozen embedding-driven classifier can be beaten by a fine-tuned LLM.\n\n## Strategy B: LoRA Fine-Tuning (QLoRA)\n\nThis strategy uses Low-Rank Adaptation (LoRA): an optimized fine-tuning algorithm that freezes the massive pre-trained weights of the model and injects trainable “adapter” weight matrices into the transformer layers. The complexity of the process boils down to training these small adapter matrices, which typically account for just 1-2% of the total model size, while maintaining 4-bit precision to save memory.\n\nThe first two steps are attaching the adapters to the base model and preparing the data for fine-tuning:\n\n``` python\nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\nfrom transformers import TrainingArguments, Trainer\nimport evaluate\n\nprint(\"1. Attaching LoRA Adapters to Base Model...\")\nmodel = prepare_model_for_kbit_training(model)\npeft_config = LoraConfig(\n    task_type=\"SEQ_CLS\",\n    r=16,\n    lora_alpha=32,\n    lora_dropout=0.05,\n    target_modules=[\"q_proj\", \"v_proj\"] # Target attention layers\n)\nlora_model = get_peft_model(model, peft_config)\nlora_model.print_trainable_parameters()\n\nprint(\"\\n2. Preparing Data for Trainer...\")\ndef tokenize_fn(examples):\n    return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True, max_length=128)\n\ntokenized_train = train_subset.map(tokenize_fn, batched=True)\ntokenized_test = test_subset.map(tokenize_fn, batched=True)\n\naccuracy_metric = evaluate.load(\"accuracy\")\ndef compute_metrics(eval_pred):\n    predictions, labels = eval_pred\n    predictions = np.argmax(predictions, axis=1)\n    return accuracy_metric.compute(predictions=predictions, references=labels)\n\n12345678910111213141516171819202122232425262728\n\nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_trainingfrom transformers import TrainingArguments, Trainerimport evaluate print(\"1. Attaching LoRA Adapters to Base Model...\")model = prepare_model_for_kbit_training(model)peft_config = LoraConfig(    task_type=\"SEQ_CLS\",    r=16,    lora_alpha=32,    lora_dropout=0.05,    target_modules=[\"q_proj\", \"v_proj\"] # Target attention layers)lora_model = get_peft_model(model, peft_config)lora_model.print_trainable_parameters() print(\"\\n2. Preparing Data for Trainer...\")def tokenize_fn(examples):    return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True, max_length=128) tokenized_train = train_subset.map(tokenize_fn, batched=True)tokenized_test = test_subset.map(tokenize_fn, batched=True) accuracy_metric = evaluate.load(\"accuracy\")def compute_metrics(eval_pred):    predictions, labels = eval_pred    predictions = np.argmax(predictions, axis=1)    return accuracy_metric.compute(predictions=predictions, references=labels)\n```\n\nNote that the result of integrating the adapter matrices into our base LLM is stored in a new model variable named `lora_model`.\n\nWe then conduct the core fine-tuning process configured as LoRA — it requires `TrainingArguments` and `Trainer` objects:\n\n```\nprint(\"\\n3. Starting QLoRA Fine-Tuning...\")\ntraining_args = TrainingArguments(\n    output_dir=\"./results\",\n    per_device_train_batch_size=8,\n    per_device_eval_batch_size=8,\n    learning_rate=2e-4,\n    num_train_epochs=3, \n    eval_strategy=\"epoch\",\n    logging_strategy=\"epoch\",\n    optim=\"paged_adamw_8bit\", # Saves VRAM by offloading optimizer states\n    fp16=True, \n    report_to=\"none\"\n)\n\ntrainer = Trainer(\n    model=lora_model,\n    args=training_args,\n    train_dataset=tokenized_train,\n    eval_dataset=tokenized_test,\n    compute_metrics=compute_metrics,\n)\n\nstart_time = time.time()\ntrainer.train()\nlora_train_time = time.time() - start_time\n\nprint(\"\\n4. Evaluating QLoRA Model...\")\neval_results = trainer.evaluate()\nlora_acc = eval_results[\"eval_accuracy\"]\n\nprint(f\"\\n--- Strategy B Results ---\")\nprint(f\"LoRA Train Time: {lora_train_time:.1f}s\")\nprint(f\"Accuracy: {lora_acc * 100:.1f}%\")\n\n123456789101112131415161718192021222324252627282930313233\n\nprint(\"\\n3. Starting QLoRA Fine-Tuning...\")training_args = TrainingArguments(    output_dir=\"./results\",    per_device_train_batch_size=8,    per_device_eval_batch_size=8,    learning_rate=2e-4,    num_train_epochs=3,     eval_strategy=\"epoch\",    logging_strategy=\"epoch\",    optim=\"paged_adamw_8bit\", # Saves VRAM by offloading optimizer states    fp16=True,     report_to=\"none\") trainer = Trainer(    model=lora_model,    args=training_args,    train_dataset=tokenized_train,    eval_dataset=tokenized_test,    compute_metrics=compute_metrics,) start_time = time.time()trainer.train()lora_train_time = time.time() - start_time print(\"\\n4. Evaluating QLoRA Model...\")eval_results = trainer.evaluate()lora_acc = eval_results[\"eval_accuracy\"] print(f\"\\n--- Strategy B Results ---\")print(f\"LoRA Train Time: {lora_train_time:.1f}s\")print(f\"Accuracy: {lora_acc * 100:.1f}%\")\n```\n\nResults (you will see more detailed results per training epoch if you execute the code):\n\n```\n--- Strategy B Results ---\nLoRA Train Time: 300.3s\nAccuracy: 87.0%\n\n123\n\n--- Strategy B Results ---LoRA Train Time: 300.3sAccuracy: 87.0%\n```\n\nClassification performance does improve when we adapt our model to the specific problem the traditional way: by fine-tuning it. That improvement comes at a cost, however — the entire process took three times as long, with all of that overhead going into the training process itself. For larger datasets, this difference in time may become substantially more pronounced.\n\n## Verdict and Conclusion\n\nFeeding frozen embeddings to an auxiliary `scikit-learn` model is the recommended approach when seeking rapid training and minimal compute overhead while avoiding backbone overfitting. QLoRA fine-tuning, by contrast, is the better choice when closing the accuracy gap is critical to your domain, or when the model needs to learn deeply embedded domain-specific nuances.\n\nThis calculus changes for purely generative tasks like text and code generation, where model adaptation — such as fine-tuning — is typically necessary for specialized behaviors. That said, the frozen embedding strategy can sometimes be a viable alternative for generative workflows in free-tier deployments where maximum performance is slightly negotiable.", "url": "https://wpnews.pro/news/fine-tuning-vs-frozen-embeddings-when-to-adapt-the-llm-vs-the-sklearn-head", "canonical_source": "https://machinelearningmastery.com/fine-tuning-vs-frozen-embeddings-when-to-adapt-the-llm-vs-the-sklearn-head/", "published_at": "2026-10-28 11:00:00+00:00", "updated_at": "2026-10-28 14:42:15.610876+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-tools", "developer-tools"], "entities": ["Qwen2.5-1.5B", "Hugging Face", "scikit-learn", "PEFT", "LoRA", "QLoRA", "ag_news", "Google Colab"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fine-tuning-vs-frozen-embeddings-when-to-adapt-the-llm-vs-the-sklearn-head", "markdown": "https://wpnews.pro/news/fine-tuning-vs-frozen-embeddings-when-to-adapt-the-llm-vs-the-sklearn-head.md", "text": "https://wpnews.pro/news/fine-tuning-vs-frozen-embeddings-when-to-adapt-the-llm-vs-the-sklearn-head.txt", "jsonld": "https://wpnews.pro/news/fine-tuning-vs-frozen-embeddings-when-to-adapt-the-llm-vs-the-sklearn-head.jsonld"}}