{"slug": "combining-llm-embeddings-with-tabular-features-in-a-unified-scikit-learn", "title": "Combining LLM Embeddings with Tabular Features in a Unified Scikit-learn Pipeline", "summary": "A tutorial demonstrates building a unified scikit-learn pipeline that combines text embeddings from Hugging Face's sentence-transformers with tabular features for classification, using the SMS Spam Collection dataset augmented with synthetic data to detect spammer users. The pipeline leverages a ColumnTransformer to run parallel preprocessing branches for text, numeric, and categorical features, and is designed to be deployment-ready.", "body_md": "In this article, you will learn how to build a unified scikit-learn pipeline that combines text embeddings generated by a lightweight open-source language model with structured tabular features for classification tasks.\n\nTopics we will cover include:\n\n- How to generate text embeddings using Hugging Face’s\n`sentence-transformers`\n\nlibrary and wrap them in a custom scikit-learn transformer class. - How to use a\n`ColumnTransformer`\n\nto run parallel preprocessing branches for text, numeric, and categorical features simultaneously. - How to assemble and evaluate a complete, deployment-ready classification pipeline on a mixed dataset combining real text data with synthetic tabular features.\n\n## Introduction\n\nReal-world tasks like ticket triage or customer churn prediction are typically addressed by building classification models. Yet, in an increasingly data-pervaded era, the data used to construct these models and perform inference on them rarely comes in a single flavor. We are often faced with a mix of tabular, structured data of numeric and qualitative nature, as well as unstructured data like text — for instance, ticket descriptions or customer messages. Feeding these data types together into machine learning models requires effective and unified pipelines that accommodate the latest data nuances and techniques to handle them.\n\nThis article shows you how to **build a clean, deployment-ready solution that encapsulates embeddings generated by open-source LLMs** (language models) **into a unified scikit-learn pipeline**, bringing together text representations and tabular features of distinct types — all based on the use of a `ColumnTransformer`\n\n. To illustrate its use, we will consider a classification scenario for detecting spammer users in a customer base.\n\n## Prerequisites\n\nInstead of resorting to a paid API like OpenAI’s or Google Gemini’s, or a massive open-source LLM like LLaMA 3, we will use a more lightweight, CPU-friendly solution to generate embeddings from a collection of texts: Hugging Face’s `sentence-transformers`\n\n. Depending on your running environment, all you may need is to install the following libraries and dependencies:\n\n```\n!pip install -q sentence-transformers scikit-learn pandas numpy\n\n1\n\n!pip install -q sentence-transformers scikit-learn pandas numpy\n```\n\nRemove the `!`\n\nif you are working in your own Python IDE rather than a cloud notebook environment like Google Colab.\n\n## Step-by-Step Guide\n\nHere’s what our intended, unified **scikit-learn pipeline architecture** looks like:\n\nBut first, we need a **mixed dataset** that looks reasonably realistic. For this, we adopt a hybrid approach: we pull a real dataset available on GitHub — the well-known SMS Spam Collection dataset containing users’ text messages labeled as spam or not — and augment it with synthetic tabular data features. Put together, the data will serve us to set up a **customer churn/triage scenario**.\n\nThe code excerpt required for data generation is a bit large, but there are plenty of comments to help you understand every decision behind the synthetic data creation process:\n\n``` python\nimport pandas as pd\nimport numpy as np\n\n# 1. Loading base text dataset from GitHub\nurl = \"https://raw.githubusercontent.com/justmarkham/pycon-2016-tutorial/master/data/sms.tsv\"\ndf = pd.read_csv(url, sep='\\t', header=None, names=['label', 'message'])\n\n# 2. Encoding original target variable first (0 for normal/ham, 1 for spam)\ndf['target'] = df['label'].map({'ham': 0, 'spam': 1})\n\n# 3. Synthesising meaningful tabular features WITH realistic overlap (noise)\n# Without noise and some degree of overlap, the classifier we will build would\n# easily achieve perfection: something not quite realistic in practice.\nnp.random.seed(42)\n\n# Account Age: Normal users can be brand new, and spammers sometimes use older hacked accounts\ndf['account_age_days'] = np.where(\n    df['target'] == 1,\n    np.random.randint(1, 365, df.shape[0]),       # Spam: 1 to 365 days\n    np.random.randint(1, 1500, df.shape[0])       # Ham: 1 to 1500 days (Massive overlap)\n)\n\n# Premium Status: Adding a bit more noise here\ndf['is_premium'] = np.where(\n    df['target'] == 1,\n    np.random.choice(['no', 'yes'], df.shape[0], p=[0.95, 0.05]), # Spam: 95% free\n    np.random.choice(['no', 'yes'], df.shape[0], p=[0.80, 0.20])  # Ham: 80% free, 20% premium\n)\n\n# Priority Score: Overlapping distributions so the model can't rely on this feature alone to classify customers\ndf['priority_score'] = np.where(\n    df['target'] == 1,\n    np.random.uniform(0.4, 1.0, df.shape[0]),   # Spam: 0.4 to 1.0\n    np.random.uniform(0.0, 0.7, df.shape[0])    # Ham: 0.0 to 0.7 (Overlap between 0.4 and 0.7)\n)\n\n# Viewing a sample of the logically cohesive mixed data\ndf.head(3)\n\n1234567891011121314151617181920212223242526272829303132333435363738\n\nimport pandas as pdimport numpy as np # 1. Loading base text dataset from GitHuburl = \"https://raw.githubusercontent.com/justmarkham/pycon-2016-tutorial/master/data/sms.tsv\"df = pd.read_csv(url, sep='\\t', header=None, names=['label', 'message']) # 2. Encoding original target variable first (0 for normal/ham, 1 for spam)df['target'] = df['label'].map({'ham': 0, 'spam': 1}) # 3. Synthesising meaningful tabular features WITH realistic overlap (noise)# Without noise and some degree of overlap, the classifier we will build would# easily achieve perfection: something not quite realistic in practice.np.random.seed(42) # Account Age: Normal users can be brand new, and spammers sometimes use older hacked accountsdf['account_age_days'] = np.where(    df['target'] == 1,    np.random.randint(1, 365, df.shape[0]),       # Spam: 1 to 365 days    np.random.randint(1, 1500, df.shape[0])       # Ham: 1 to 1500 days (Massive overlap)) # Premium Status: Adding a bit more noise heredf['is_premium'] = np.where(    df['target'] == 1,    np.random.choice(['no', 'yes'], df.shape[0], p=[0.95, 0.05]), # Spam: 95% free    np.random.choice(['no', 'yes'], df.shape[0], p=[0.80, 0.20])  # Ham: 80% free, 20% premium) # Priority Score: Overlapping distributions so the model can't rely on this feature alone to classify customersdf['priority_score'] = np.where(    df['target'] == 1,    np.random.uniform(0.4, 1.0, df.shape[0]),   # Spam: 0.4 to 1.0    np.random.uniform(0.0, 0.7, df.shape[0])    # Ham: 0.0 to 0.7 (Overlap between 0.4 and 0.7)) # Viewing a sample of the logically cohesive mixed datadf.head(3)\n```\n\nExample output:\n\nThe next step is key, as this is where we **create the custom text transformer** — see the leftmost branch in the previous diagram. In scikit-learn, this is done by creating a custom class that inherits from `TransformerMixin`\n\nand `BaseEstimator`\n\n. The requirement is to define `fit()`\n\nand `transform()`\n\nmethods, just like any pre-existing data transformation class in the library (e.g. standard scalers and one-hot encoders).\n\n``` python\nfrom sklearn.base import BaseEstimator, TransformerMixin\nfrom sentence_transformers import SentenceTransformer\n\nclass TextEmbedder(BaseEstimator, TransformerMixin):\n    def __init__(self, model_name='all-MiniLM-L6-v2'):\n        self.model_name = model_name\n        self.model = None\n\n    def fit(self, X, y=None):\n        # Initializing the model in fit() to comply with sklearn cloning rules\n        if self.model is None:\n            self.model = SentenceTransformer(self.model_name)\n        return self\n        \n    def transform(self, X, y=None):\n        # Handling pandas DataFrame (extract the first column as a list of strings)\n        if isinstance(X, pd.DataFrame):\n            texts = X.iloc[:, 0].astype(str).tolist()\n        else:\n            texts = pd.Series(X).astype(str).tolist()\n            \n        # Using the specified LLM, generate and return embeddings as a 2D numpy array\n        return self.model.encode(texts, show_progress_bar=False)\n\n1234567891011121314151617181920212223\n\nfrom sklearn.base import BaseEstimator, TransformerMixinfrom sentence_transformers import SentenceTransformer class TextEmbedder(BaseEstimator, TransformerMixin):    def __init__(self, model_name='all-MiniLM-L6-v2'):        self.model_name = model_name        self.model = None     def fit(self, X, y=None):        # Initializing the model in fit() to comply with sklearn cloning rules        if self.model is None:            self.model = SentenceTransformer(self.model_name)        return self            def transform(self, X, y=None):        # Handling pandas DataFrame (extract the first column as a list of strings)        if isinstance(X, pd.DataFrame):            texts = X.iloc[:, 0].astype(str).tolist()        else:            texts = pd.Series(X).astype(str).tolist()                    # Using the specified LLM, generate and return embeddings as a 2D numpy array        return self.model.encode(texts, show_progress_bar=False)\n```\n\nNotice that we specify the Hugging Face sentence-transformer model to use — namely `all-MiniLM-L6-v2`\n\n— in the constructor method, and call the model in `transform()`\n\nto map texts into embeddings.\n\nNext, once we have our embeddings, we apply the parallel data preprocessing required by the other features. Since this relies entirely on already-implemented classes in scikit-learn, we can directly assemble all the type-specific preprocessing steps into an overarching, unified pipeline. We distinguish numerical columns from categorical ones, applying standard scaling to the former and one-hot encoding to the latter. Together with the previously implemented text embedding step, this gives us three processing branches that run in parallel. The way to implement this is through a `ColumnTransformer`\n\nobject that contains a list of three “processing branches.” This mechanism keeps the whole dataset together, without the need to manually split and re-unify features.\n\nAfter that, we add the final stage: a random forest classifier. The entire process looks as follows:\n\n``` python\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import StandardScaler, OneHotEncoder\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import classification_report\n\n# Split data\nX = df[['message', 'account_age_days', 'priority_score', 'is_premium']]\ny = df['target']\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\n# Define column groups\ntext_features = ['message']\nnumeric_features = ['account_age_days', 'priority_score']\ncategorical_features = ['is_premium']\n\n# Build the ColumnTransformer\npreprocessor = ColumnTransformer(\n    transformers=[\n        ('text', TextEmbedder(), text_features),\n        ('num', StandardScaler(), numeric_features),\n        ('cat', OneHotEncoder(handle_unknown='ignore'), categorical_features)\n    ],\n    remainder='drop' # Drop any columns not explicitly defined\n)\n\n# Assemble the final pipeline\npipeline = Pipeline(steps=[\n    ('preprocessor', preprocessor),\n    ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))\n])\n\n1234567891011121314151617181920212223242526272829303132\n\nfrom sklearn.compose import ColumnTransformerfrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import StandardScaler, OneHotEncoderfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import classification_report # Split dataX = df[['message', 'account_age_days', 'priority_score', 'is_premium']]y = df['target']X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Define column groupstext_features = ['message']numeric_features = ['account_age_days', 'priority_score']categorical_features = ['is_premium'] # Build the ColumnTransformerpreprocessor = ColumnTransformer(    transformers=[        ('text', TextEmbedder(), text_features),        ('num', StandardScaler(), numeric_features),        ('cat', OneHotEncoder(handle_unknown='ignore'), categorical_features)    ],    remainder='drop' # Drop any columns not explicitly defined) # Assemble the final pipelinepipeline = Pipeline(steps=[    ('preprocessor', preprocessor),    ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))])\n```\n\nNow that we have assembled the entire pipeline, it’s time to try it out! The final piece of code trains the model — a process that, thanks to the pipeline encapsulation, implicitly carries out all the preceding data preparations — and evaluates it on the test set we set aside earlier:\n\n```\n# Training the model (this will take a moment to download the HF model and embed the texts)\nprint(\"Training pipeline...\")\npipeline.fit(X_train, y_train)\n\n# Evaluating on test examples\nprint(\"Predicting and evaluating...\")\ny_pred = pipeline.predict(X_test)\nprint(classification_report(y_test, y_pred))\n\n12345678\n\n# Training the model (this will take a moment to download the HF model and embed the texts)print(\"Training pipeline...\")pipeline.fit(X_train, y_train) # Evaluating on test examplesprint(\"Predicting and evaluating...\")y_pred = pipeline.predict(X_test)print(classification_report(y_test, y_pred))\n```\n\nResults:\n\n```\nPredicting and evaluating...\n              precision    recall  f1-score   support\n\n           0       0.99      1.00      0.99       966\n           1       1.00      0.91      0.95       149\n\n    accuracy                           0.99      1115\n   macro avg       0.99      0.95      0.97      1115\nweighted avg       0.99      0.99      0.99      1115\n\n123456789\n\nPredicting and evaluating...              precision    recall  f1-score   support            0       0.99      1.00      0.99       966           1       1.00      0.91      0.95       149     accuracy                           0.99      1115   macro avg       0.99      0.95      0.97      1115weighted avg       0.99      0.99      0.99      1115\n```\n\nThese results are pretty decent. Part of the reason is that the real dataset used for the labeled texts is known for being easily class-separable and therefore not hard to classify with high accuracy. We also intentionally added noise and overlap when creating the other synthetic attributes to introduce a bit of challenge for our classifier — otherwise, it might have achieved 100% accuracy, which would not be very informative.\n\n## Conclusion\n\nThis article tackled an increasingly common problem in the AI and data science landscape: leveraging text data and combining it with structured data features traditionally fed to downstream machine learning models for predictive tasks like classification. We used scikit-learn’s transformer classes and a pre-trained language model to build a unified pipeline that cleanly and elegantly processes these mixed data types, yielding a robust and easily reusable solution.", "url": "https://wpnews.pro/news/combining-llm-embeddings-with-tabular-features-in-a-unified-scikit-learn", "canonical_source": "https://machinelearningmastery.com/combining-llm-embeddings-with-tabular-features-in-a-unified-scikit-learn-pipeline/", "published_at": "2026-08-31 12:00:18+00:00", "updated_at": "2026-08-31 12:54:06.523663+00:00", "lang": "en", "topics": ["machine-learning", "natural-language-processing", "developer-tools"], "entities": ["Hugging Face", "sentence-transformers", "scikit-learn", "SMS Spam Collection"], "alternates": {"html": "https://wpnews.pro/news/combining-llm-embeddings-with-tabular-features-in-a-unified-scikit-learn", "markdown": "https://wpnews.pro/news/combining-llm-embeddings-with-tabular-features-in-a-unified-scikit-learn.md", "text": "https://wpnews.pro/news/combining-llm-embeddings-with-tabular-features-in-a-unified-scikit-learn.txt", "jsonld": "https://wpnews.pro/news/combining-llm-embeddings-with-tabular-features-in-a-unified-scikit-learn.jsonld"}}