Multilingual Text Classification with Scikit-LLM and Multilingual Embeddings

A tutorial published on Scikit-LLM shows developers how to build a multilingual text classification pipeline using multilingual LLM embeddings and scikit-learn without training a separate model per language. The walkthrough configures a free, local embedding pipeline with Ollama and the open-source BGE-M3 model, then trains and evaluates a logistic regression classifier on top of those embeddings using a real-world review dataset. The setup installs scikit-llm and datasets version 2.19.1, points Scikit-LLM at a local Ollama server via SKLLMConfig.set_gpt_url("http://localhost:11434/v1/"), and uses a dummy API key to avoid paid services such as OpenAI.

In this article, you will learn how to build a multilingual text classification pipeline using multilingual large language model LLM embeddings and Scikit-learn, without training separate models for each language. Topics we will cover include: - What multilingual LLM embeddings are and why they eliminate the need for language-specific models. - How to set up a free, local embedding pipeline using Ollama, BGE-M3, and Scikit-LLM. - How to train and evaluate a logistic regression classifier on top of multilingual embeddings using a real-world review dataset. Introduction Building machine learning models for a global audience , such as text classifiers based on multilingual data , traditionally required training a separate model for each language. Thus, the process could easily become unmanageable. Thankfully, progress in LLMs also extends to scenarios like this Multilingual LLM embeddings are numerical representations of text produced by a model that maps text from different languages into a common vector space. With these “barrier-free” embeddings, all it takes thereafter is training a downstream, lightweight classifier on top of them. Let’s uncover how to do this step-by-step, aided by Scikit-LLM. Initial Setup In the sequel, we will construct a multilingual text classification pipeline aided by Scikit-LLM and scikit-learn. Installing Python dependencies pip install scikit-llm "datasets==2.19.1" -q Fix Colab's missing system dependencies first version-dependent, use with care in other environments apt-get update -qq && apt-get install -y -qq zstd Installing Ollama distribution curl -fsSL https://ollama.com/install.sh | sh 12345678 Installing Python dependenciespip install scikit-llm "datasets==2.19.1" -q Fix Colab's missing system dependencies first version-dependent, use with care in other environments apt-get update -qq && apt-get install -y -qq zstd Installing Ollama distributioncurl -fsSL https://ollama.com/install.sh | sh Ensuring a 100% free and runnable solution in a variety of running environments, including notebooks, requires bypassing paid APIs like OpenAI. That’s why, instead, we have installed an Ollama distribution offering a variety of free LLMs. Accordingly, in the next steps we will configure Scikit-LLM to speak to a local Ollama server running BGE-M3 , which is a state-of-the-art, open-source model supporting multilingual information in the embedding generation process. Next, we start the Ollama server as a background process —this is the most hassle-free way to use Ollama in a cloud-based notebook, but not mandatory if working with your own IDE and local Ollama distribution. We also pull the aforementioned multilingual model for embedding generation, BGE-M3 more information about this model on its official website https://bge-model.com/bge/bge m3.html . python import subprocess import time Starting the Ollama server in the background subprocess.Popen "ollama", "serve" time.sleep 5 Give the server a few seconds to initialize Pulling the multilingual embedding model ollama pull bge-m3 123456789 import subprocessimport time Starting the Ollama server in the backgroundsubprocess.Popen "ollama", "serve" time.sleep 5 Give the server a few seconds to initialize Pulling the multilingual embedding modelollama pull bge-m3 The last configuration step is to use Scikit-LLM’s configuration module to point it to our Ollama instance. The configuration approach we are using does not require an actual key, but a dummy one, as shown below: python from skllm.config import SKLLMConfig Point Scikit-LLM to our local Ollama instance SKLLMConfig.set gpt url "http://localhost:11434/v1/" Provide a dummy key required by the internal client, but safely ignored by Ollama SKLLMConfig.set openai key "free-friendly-dummy-key" 1234567 from skllm.config import SKLLMConfig Point Scikit-LLM to our local Ollama instanceSKLLMConfig.set gpt url "http://localhost:11434/v1/" Provide a dummy key required by the internal client, but safely ignored by Ollama SKLLMConfig.set openai key "free-friendly-dummy-key" Building the Pipeline The first major step in building our multilingual classification pipeline is, of course, getting the data. We will consider the Amazon Multi-language Reviews dataset, which has labeled customer reviews on a 5-star rating scale internally encoded with labels 0 to 4 . To avoid an overly time-consuming execution — especially regarding the embedding generation process later on — we will load a total of 2000 reviews in both English and Spanish. Feel free to select a larger sample if you’d like to, but try to keep it language-balanced and ensure random shuffling of your data before applying further steps like a training-test split. python from datasets import load dataset import pandas as pd print "Loading and shuffling data to ensure class diversity..." 1. Loading the complete split 2. Shuffling it randomly with shuffle 3. Extracting 1000 varied samples with select range 1000 data en = load dataset "mteb/amazon reviews multi", "en", split="train", trust remote code=True .shuffle seed=42 .select range 1000 data es = load dataset "mteb/amazon reviews multi", "es", split="train", trust remote code=True .shuffle seed=42 .select range 1000 Combining into a single DataFrame df = pd.concat pd.DataFrame data en , pd.DataFrame data es , ignore index=True Shuffling bilingual data df = df.sample frac=1, random state=42 .reset index drop=True Features and Labels X = df 'text' y = df 'label' print f"Total samples: {len X }" print "\n--- Class Verification should have samples from 0 to 4 ---" print y.value counts 1234567891011121314151617181920212223242526272829 from datasets import load datasetimport pandas as pd print "Loading and shuffling data to ensure class diversity..." 1. Loading the complete split 2. Shuffling it randomly with shuffle 3. Extracting 1000 varied samples with select range 1000 data en = load dataset "mteb/amazon reviews multi", "en", split="train", trust remote code=True .shuffle seed=42 .select range 1000 data es = load dataset "mteb/amazon reviews multi", "es", split="train", trust remote code=True .shuffle seed=42 .select range 1000 Combining into a single DataFramedf = pd.concat pd.DataFrame data en , pd.DataFrame data es , ignore index=True Shuffling bilingual datadf = df.sample frac=1, random state=42 .reset index drop=True Features and LabelsX = df 'text' y = df 'label' print f"Total samples: {len X }" print "\n--- Class Verification should have samples from 0 to 4 ---" print y.value counts Output: Loading and shuffling data to ensure class diversity... Total samples: 2000 --- Class Verification should have samples from 0 to 4 --- label 0 444 3 410 2 404 4 380 1 362 Name: count, dtype: int64 1234567891011 Loading and shuffling data to ensure class diversity...Total samples: 2000 --- Class Verification should have samples from 0 to 4 ---label0 4443 4102 4044 3801 362Name: count, dtype: int64 The magic happens next. We define a scikit-learn pipeline consisting of two major stages: - Using a GPTVectorizer from Scikit-LLM and having it set up to utilize our previously loaded BGE-M3 model for building embeddings. - Feeding the embeddings to train a classifier based on a LogisticRegression model type. python from skllm.models.gpt.vectorization import GPTVectorizer from sklearn.pipeline import Pipeline from sklearn.linear model import LogisticRegression from sklearn.model selection import train test split from sklearn.metrics import classification report Splitting into 80% training and 20% testing X train, X test, y train, y test = train test split X, y, test size=0.2, random state=42 Defining the Pipeline pipeline = Pipeline "vectorizer", GPTVectorizer model="bge-m3", batch size=32 , "classifier", LogisticRegression max iter=1000, random state=42 Training the pipeline print "Extracting embeddings and training classifier..." pipeline.fit X train, y train 123456789101112131415161718 from skllm.models.gpt.vectorization import GPTVectorizerfrom sklearn.pipeline import Pipelinefrom sklearn.linear model import LogisticRegressionfrom sklearn.model selection import train test splitfrom sklearn.metrics import classification report Splitting into 80% training and 20% testingX train, X test, y train, y test = train test split X, y, test size=0.2, random state=42 Defining the Pipelinepipeline = Pipeline "vectorizer", GPTVectorizer model="bge-m3", batch size=32 , "classifier", LogisticRegression max iter=1000, random state=42 Training the pipelineprint "Extracting embeddings and training classifier..." pipeline.fit X train, y train Why did I say the magic takes place here? Let’s look more closely: BGE-M3 is a multilingual embedding model that has been pre-trained on massive data spanning over 100 languages. Put another way, it is capable of internally mapping both our English and Spanish reviews into a common dimensional embedding space: not based on their concrete vocabulary, but based on the meaning behind it. Thus, language barriers disappear during the process of generating embeddings, with LLM outputs for “This product is fantastic ” and “¡Este producto es fantástico ” being nearly identical. As a result, by the time the embeddings arrive at the logistic regression model for training and inference, the classifier doesn’t actually care about the language anymore. It has the information it needs to perform rating classifications on product reviews. print "Evaluating on the test set..." y pred = pipeline.predict X test print "\n--- Classification Report ---" print classification report y test, y pred 12345 print "Evaluating on the test set..." y pred = pipeline.predict X test print "\n--- Classification Report ---" print classification report y test, y pred Results: --- Classification Report --- precision recall f1-score support 0 0.66 0.78 0.72 82 1 0.40 0.30 0.34 64 2 0.46 0.46 0.46 91 3 0.56 0.54 0.55 84 4 0.71 0.73 0.72 79 accuracy 0.57 400 macro avg 0.56 0.56 0.56 400 weighted avg 0.56 0.57 0.56 400 123456789101112 --- Classification Report --- precision recall f1-score support 0 0.66 0.78 0.72 82 1 0.40 0.30 0.34 64 2 0.46 0.46 0.46 91 3 0.56 0.54 0.55 84 4 0.71 0.73 0.72 79 accuracy 0.57 400 macro avg 0.56 0.56 0.56 400weighted avg 0.56 0.57 0.56 400 The results are just okay, but not great. There is significantly better performance in correctly predicting extreme ratings 0 for 1-star, 4 for 5-star than for predicting intermediate ratings. Don’t panic; there are at least two reasons for this: 1. The classification task at hand is inherently challenging: distinguishing between a 3-star and a 4-star review is intuitively harder than discerning, for instance, between positive, negative, and neutral reviews. 2. More importantly, we have used just 2000 samples 80% of them for model training , but these samples are embeddings with 1024 features each. Feeding such a small amount of high-dimensional data to a classifier is most likely the perfect recipe for overfitting your model. If you have the time to run the code for longer, try using a few thousand more examples instead. Wrapping Up In traditional natural language processing, we were often faced with two far-from-ideal options when handling multilingual data for predictive tasks like text classification: translate all your data into a base language — a slow, expensive process with frequent loss of nuance — or train separate models: one for every language. In the pipeline we just built, the heavy burden is assumed by the multilingual embedding model BGE-M3 leveraged through Scikit-LLM, which is capable of transparently mapping text across a variety of languages into a uniform embedding space.