Serving Scikit-LLM Pipelines as REST APIs with FastAPI A tutorial demonstrates serving a Scikit-LLM zero-shot text classification pipeline backed by a local Ollama llama3 model as an asynchronous REST API built with FastAPI, avoiding paid cloud LLM APIs. The walkthrough configures Scikit-LLM via SKLLMConfig.set_gpt_url to point at http://localhost:11434/v1/ with a dummy key, builds a ZeroShotGPTClassifier using scikit-learn's fit/predict syntax, serializes it to scikit_llm_model.joblib with joblib, and exposes it through a FastAPI endpoint tested from the interactive docs page. The stack requires Python 3.11 plus the scikit-llm, fastapi, uvicorn, joblib and pydantic packages. In this article, you will learn how to serve a Scikit-LLM zero-shot text classification pipeline, powered by a local Ollama model, as a REST API with FastAPI. Topics we will cover include: - How to configure Scikit-LLM to use a local Ollama model instead of a paid cloud API. - How to build and serialize a zero-shot classifier using the familiar scikit-learn fit/predict syntax. - How to serve the classifier through an asynchronous FastAPI endpoint and test it from the interactive docs page. Introduction It looks like Scikit-LLM is here to stay. This innovative Python framework pushes the boundaries of machine learning and LLMs by treating them as two sides of the same coin, enabling scikit-learn-based pipelines fueled by powerful pre-trained LLMs adapted to perform specific tasks — for instance, building a smart zero-shot classification engine. All of this sounds enthralling until you try to put it into production and instantly get chained to costly cloud APIs, latency spikes, and so on. This article covers the first crucial steps toward deploying a deep semantic reasoning pipeline fueled by an LLM and combined with the reliability of standardized machine learning syntax — enter the .fit and .predict methods — as well as predictable outputs like those found in traditional machine learning pipelines. We will put together an open-source stack composed of Scikit-LLM , FastAPI , and Ollama , teaching you how to wire up a local, self-hosted LLM that acts as a deterministic text labeler: all wrapped in an asynchronous, blazing-fast REST API, thereby bypassing cloud costs and commercial lock-in. Setup and Initial Configurations For this practical demonstration, I have used Visual Studio Code with a virtual environment using Python 3.11. I have also installed some basic libraries like pandas and scikit-learn in the virtual environment. In addition, you’ll need to install the following specialized libraries to make the code work: pip install scikit-llm fastapi uvicorn joblib pydantic 1 pip install scikit-llm fastapi uvicorn joblib pydantic You’ll also need to install Ollama and pull locally download llama3 . This article explains the process to install Ollama in more detail, but chances are you won’t need it if you just visit the Ollama website https://ollama.com/download : it’s pretty simple to install, I promise Building, Deploying, and Testing the Pipeline Once everything is correctly installed and you have resolved any possible issues caused by version compatibility with your Python interpreter, let’s move on to the core of the process. We’ll create a new project folder in our IDE and create two Python files inside it, which will contain the code: name them train.py and app.py . Let’s start with train.py . In this Python file, we will add the logic to train and serialize the Scikit-LLM pipeline. To keep things simple but scalable, we will just invoke the Ollama model and attach it to a one-stage pipeline consisting of a zero-shot classifier — in simple terms, a pre-trained LLM turned into a classifier by giving it the set of possible categories, or labels, from an example dataset of texts, without actual model training. After that, we serialize the pipeline: once you run the code below, a file named scikit llm model.joblib containing the serialized pipeline will be created in your project folder. Here’s the code to do this: python import joblib from skllm.config import SKLLMConfig from skllm.models.gpt.classification.zero shot import ZeroShotGPTClassifier 1. Routing Scikit-LLM to local Ollama instead of cloud APIs usually paid SKLLMConfig.set gpt url "http://localhost:11434/v1/" SKLLMConfig.set openai key "ollama-dummy" X = "The battery drains too fast", "I love the new camera", "Screen is completely cracked" y = "Battery Issue", "Positive Feedback", "Hardware Damage" 2. Appending the specific Ollama model name after custom url:: clf = ZeroShotGPTClassifier model="custom url::llama3" clf.fit X, y 3. Serializing the estimator joblib.dump clf, "scikit llm model.joblib" print "Pipeline serialized successfully." 123456789101112131415161718 import joblibfrom skllm.config import SKLLMConfigfrom skllm.models.gpt.classification.zero shot import ZeroShotGPTClassifier 1. Routing Scikit-LLM to local Ollama instead of cloud APIs usually paid SKLLMConfig.set gpt url "http://localhost:11434/v1/" SKLLMConfig.set openai key "ollama-dummy" X = "The battery drains too fast", "I love the new camera", "Screen is completely cracked" y = "Battery Issue", "Positive Feedback", "Hardware Damage" 2. Appending the specific Ollama model name after custom url::clf = ZeroShotGPTClassifier model="custom url::llama3" clf.fit X, y 3. Serializing the estimatorjoblib.dump clf, "scikit llm model.joblib" print "Pipeline serialized successfully." The next step is to add the code for the app.py file. Run this code after generating the serialized pipeline. This file is needed to: 1. Reset and re-apply the Scikit-LLM configuration needed for the serving environment. 2. Create a FastAPI app instance and define the predict endpoint with the necessary logic to enable inference by the model once deployed. Note that the service can take multiple input texts in a single call, thus returning multiple class predictions in one shot. 3. Start the server using uvicorn , which the final lines of code take care of. python from contextlib import asynccontextmanager import joblib import asyncio from fastapi import FastAPI, HTTPException from pydantic import BaseModel from skllm.config import SKLLMConfig Applying configuration for the serving environment SKLLMConfig.set gpt url "http://localhost:11434/v1/" SKLLMConfig.set openai key "ollama-dummy" ml models = {} @asynccontextmanager async def lifespan app: FastAPI : Loading model into memory on startup ml models "classifier" = joblib.load "scikit llm model.joblib" yield ml models.clear app = FastAPI lifespan=lifespan class InferenceRequest BaseModel : texts: list str @app.post "/predict" async def predict request: InferenceRequest : clf = ml models "classifier" try: Offload blocking Scikit-LLM I/O to a background thread predictions = await asyncio.to thread clf.predict, request.texts return {"predictions": predictions.tolist } except Exception as e: raise HTTPException status code=500, detail=str e if name == " main ": import uvicorn Starts the FastAPI server when you run 'python app.py' uvicorn.run app, host="127.0.0.1", port=8000 123456789101112131415161718192021222324252627282930313233343536373839404142 from contextlib import asynccontextmanagerimport joblibimport asynciofrom fastapi import FastAPI, HTTPExceptionfrom pydantic import BaseModelfrom skllm.config import SKLLMConfig Applying configuration for the serving environmentSKLLMConfig.set gpt url "http://localhost:11434/v1/" SKLLMConfig.set openai key "ollama-dummy" ml models = {} @asynccontextmanagerasync def lifespan app: FastAPI : Loading model into memory on startup ml models "classifier" = joblib.load "scikit llm model.joblib" yield ml models.clear app = FastAPI lifespan=lifespan class InferenceRequest BaseModel : texts: list str @app.post "/predict" async def predict request: InferenceRequest : clf = ml models "classifier" try: Offload blocking Scikit-LLM I/O to a background thread predictions = await asyncio.to thread clf.predict, request.texts return {"predictions": predictions.tolist } except Exception as e: raise HTTPException status code=500, detail=str e if name == " main ": import uvicorn Starts the FastAPI server when you run 'python app.py' uvicorn.run app, host="127.0.0.1", port=8000 Once you run this code in your IDE, if everything goes smoothly, you should see the following messages in the output console: INFO: Started server process 9351 INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8000 Press CTRL+C to quit 1234 INFO: Started server process 9351 INFO: Waiting for application startup.INFO: Application startup complete.INFO: Uvicorn running on http://127.0.0.1:8000 Press CTRL+C to quit Now your pipeline is accessible well, locally accessible, I mean , and you can open the FastAPI interface by browsing to http://127.0.0.1:8000/docs http://127.0.0.1:8000/docs . To test the app, click the “ Try it out ” button inside the expandable “ /predict ” block, and edit the “Request body” text box to paste the following input examples or any of your own: { "texts": "My phone dies by noon even if I don't use it.", "I dropped it on the sidewalk and the glass shattered.", "This is the best device I have ever purchased " } 1234567 { "texts": "My phone dies by noon even if I don't use it.", "I dropped it on the sidewalk and the glass shattered.", "This is the best device I have ever purchased " } Then click “Execute”, and your pipeline’s LLM will start doing its magic. In a few seconds, the predictions will pop up: { "predictions": "Battery Issue", "Hardware Damage", "Positive Feedback" } 1234567 { "predictions": "Battery Issue", "Hardware Damage", "Positive Feedback" } Final Remarks Congratulations if you made it here Just be aware of a few notes about what we have just done together: - The classifier we built and deployed relies on the label set y we passed to it in the fit method early on. We didn’t update the LLM’s weights this process would have taken considerably longer . Instead, the classifier stored the specific closed-domain set of categories to choose from when performing inference on texts, just as if we had written a multiple-choice question in a prompt. - For this reason, when turning a pre-trained model into a zero-shot classifier, we are restricting its “view of the world”, forcing it to map any input no matter what to exactly one of the labels in the label set. In exchange for this closed-world view, we add a layer of reliability and predictability to an LLM that already had a profound level of knowledge. - We built an extremely simple, one-stage pipeline, but thanks to Scikit-LLM’s compatibility with scikit-learn objects and syntax, we could have created a larger pipeline with additional preprocessing steps, similar to the example in this related article https://machinelearningmastery.com/text-summarization-with-scikit-llm/ .