# Serving Scikit-LLM Pipelines as REST APIs with FastAPI

> Source: <https://machinelearningmastery.com/serving-scikit-llm-pipelines-as-rest-apis-with-fastapi/>
> Published: 2026-11-16 11:00:00+00:00

In this article, you will learn how to serve a Scikit-LLM zero-shot text classification pipeline, powered by a local Ollama model, as a REST API with FastAPI.

Topics we will cover include:

- How to configure Scikit-LLM to use a local Ollama model instead of a paid cloud API.
- How to build and serialize a zero-shot classifier using the familiar scikit-learn fit/predict syntax.
- How to serve the classifier through an asynchronous FastAPI endpoint and test it from the interactive docs page.

## Introduction

It looks like **Scikit-LLM** is here to stay. This innovative Python framework pushes the boundaries of machine learning and LLMs by treating them as two sides of the same coin, enabling scikit-learn-based pipelines fueled by powerful pre-trained LLMs adapted to perform specific tasks — for instance, building a smart zero-shot classification engine. All of this sounds enthralling until you try to put it into production and instantly get chained to costly cloud APIs, latency spikes, and so on.

This article covers the first crucial steps toward deploying a deep semantic reasoning pipeline fueled by an LLM and combined with the reliability of standardized machine learning syntax — enter the `.fit()` and `.predict()` methods — as well as predictable outputs like those found in traditional machine learning pipelines.

We will put together an open-source stack composed of **Scikit-LLM**, **FastAPI**, and **Ollama**, teaching you how to wire up a local, self-hosted LLM that acts as a deterministic text labeler: all wrapped in an asynchronous, blazing-fast REST API, thereby bypassing cloud costs and commercial lock-in.

## Setup and Initial Configurations

For this practical demonstration, I have used **Visual Studio Code** with a virtual environment using Python 3.11. I have also installed some basic libraries like pandas and scikit-learn in the virtual environment. In addition, you’ll need to install the following specialized libraries to make the code work:

```
pip install scikit-llm fastapi uvicorn joblib pydantic

1

pip install scikit-llm fastapi uvicorn joblib pydantic
```

You’ll also need to install Ollama and pull (locally download) `llama3`. This article explains the process to install Ollama in more detail, but chances are you won’t need it if you just visit the [Ollama website](https://ollama.com/download): it’s pretty simple to install, I promise!

## Building, Deploying, and Testing the Pipeline

Once everything is correctly installed and you have resolved any possible issues caused by version compatibility with your Python interpreter, let’s move on to the core of the process. We’ll create a new project folder in our IDE and create two Python files inside it, which will contain the code: name them `train.py` and `app.py`.

Let’s start with `train.py`. In this Python file, we will add the logic to train and serialize the Scikit-LLM pipeline. To keep things simple but scalable, we will just invoke the Ollama model and attach it to a one-stage pipeline consisting of a zero-shot classifier — in simple terms, a pre-trained LLM turned into a classifier by giving it the set of possible categories, or labels, from an example dataset of texts, without actual model training.

After that, we serialize the pipeline: once you run the code below, a file named `scikit_llm_model.joblib` containing the serialized pipeline will be created in your project folder. Here’s the code to do this:

``` python
import joblib
from skllm.config import SKLLMConfig
from skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier

# 1. Routing Scikit-LLM to local Ollama instead of cloud APIs (usually paid)
SKLLMConfig.set_gpt_url("http://localhost:11434/v1/")
SKLLMConfig.set_openai_key("ollama-dummy")

X = ["The battery drains too fast", "I love the new camera", "Screen is completely cracked"]
y = ["Battery Issue", "Positive Feedback", "Hardware Damage"]

# 2. Appending the specific Ollama model name after custom_url::
clf = ZeroShotGPTClassifier(model="custom_url::llama3")
clf.fit(X, y)

# 3. Serializing the estimator
joblib.dump(clf, "scikit_llm_model.joblib")
print("Pipeline serialized successfully.")

123456789101112131415161718

import joblibfrom skllm.config import SKLLMConfigfrom skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier # 1. Routing Scikit-LLM to local Ollama instead of cloud APIs (usually paid)SKLLMConfig.set_gpt_url("http://localhost:11434/v1/")SKLLMConfig.set_openai_key("ollama-dummy") X = ["The battery drains too fast", "I love the new camera", "Screen is completely cracked"]y = ["Battery Issue", "Positive Feedback", "Hardware Damage"] # 2. Appending the specific Ollama model name after custom_url::clf = ZeroShotGPTClassifier(model="custom_url::llama3")clf.fit(X, y) # 3. Serializing the estimatorjoblib.dump(clf, "scikit_llm_model.joblib")print("Pipeline serialized successfully.")
```

The next step is to add the code for the `app.py` file. Run this code after generating the serialized pipeline. This file is needed to:

1. Reset and re-apply the Scikit-LLM configuration needed for the serving environment.
2. Create a FastAPI app instance and define the `predict()` endpoint with the necessary logic to enable inference by the model once deployed. Note that the service can take multiple input texts in a single call, thus returning multiple class predictions in one shot.
3. Start the server using `uvicorn` , which the final lines of code take care of.

``` python
from contextlib import asynccontextmanager
import joblib
import asyncio
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from skllm.config import SKLLMConfig

# Applying configuration for the serving environment
SKLLMConfig.set_gpt_url("http://localhost:11434/v1/")
SKLLMConfig.set_openai_key("ollama-dummy")

ml_models = {}

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Loading model into memory on startup
    ml_models["classifier"] = joblib.load("scikit_llm_model.joblib")
    yield
    ml_models.clear()

app = FastAPI(lifespan=lifespan)

class InferenceRequest(BaseModel):
    texts: list[str]

@app.post("/predict")
async def predict(request: InferenceRequest):
    clf = ml_models["classifier"]

    try:
        # Offload blocking Scikit-LLM I/O to a background thread
        predictions = await asyncio.to_thread(clf.predict, request.texts)
        return {"predictions": predictions.tolist()}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

if __name__ == "__main__":
    import uvicorn
    # Starts the FastAPI server when you run 'python app.py'
    uvicorn.run(app, host="127.0.0.1", port=8000)

123456789101112131415161718192021222324252627282930313233343536373839404142

from contextlib import asynccontextmanagerimport joblibimport asynciofrom fastapi import FastAPI, HTTPExceptionfrom pydantic import BaseModelfrom skllm.config import SKLLMConfig # Applying configuration for the serving environmentSKLLMConfig.set_gpt_url("http://localhost:11434/v1/")SKLLMConfig.set_openai_key("ollama-dummy") ml_models = {} @asynccontextmanagerasync def lifespan(app: FastAPI):    # Loading model into memory on startup    ml_models["classifier"] = joblib.load("scikit_llm_model.joblib")    yield    ml_models.clear() app = FastAPI(lifespan=lifespan) class InferenceRequest(BaseModel):    texts: list[str] @app.post("/predict")async def predict(request: InferenceRequest):    clf = ml_models["classifier"]     try:        # Offload blocking Scikit-LLM I/O to a background thread        predictions = await asyncio.to_thread(clf.predict, request.texts)        return {"predictions": predictions.tolist()}    except Exception as e:        raise HTTPException(status_code=500, detail=str(e))   if __name__ == "__main__":    import uvicorn    # Starts the FastAPI server when you run 'python app.py'    uvicorn.run(app, host="127.0.0.1", port=8000)
```

Once you run this code in your IDE, if everything goes smoothly, you should see the following messages in the output console:

```
INFO:     Started server process [9351]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

1234

INFO:     Started server process [9351]INFO:     Waiting for application startup.INFO:     Application startup complete.INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
```

Now your pipeline is accessible (well, locally accessible, I mean), and you can open the FastAPI interface by browsing to [http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs). To test the app, click the “**Try it out**” button inside the expandable “**/predict**” block, and edit the “Request body” text box to paste the following input examples or any of your own:

```
{
  "texts": [
    "My phone dies by noon even if I don't use it.",
    "I dropped it on the sidewalk and the glass shattered.",
    "This is the best device I have ever purchased!"
  ]
}

1234567

{  "texts": [    "My phone dies by noon even if I don't use it.",    "I dropped it on the sidewalk and the glass shattered.",    "This is the best device I have ever purchased!"  ]}
```

Then click “Execute”, and your pipeline’s LLM will start doing its magic. In a few seconds, the predictions will pop up:

```
{
  "predictions": [
    "Battery Issue",
    "Hardware Damage",
    "Positive Feedback"
  ]
}

1234567

{  "predictions": [    "Battery Issue",    "Hardware Damage",    "Positive Feedback"  ]}
```

## Final Remarks

Congratulations if you made it here! Just be aware of a few notes about what we have just done together:

- The classifier we built and deployed relies on the label set `y` we passed to it in the`fit()` method early on. We didn’t update the LLM’s weights (this process would have taken considerably longer!). Instead, the classifier stored the specific closed-domain set of categories to choose from when performing inference on texts, just as if we had written a multiple-choice question in a prompt.
- For this reason, when turning a pre-trained model into a zero-shot classifier, we are restricting its “view of the world”, forcing it to map any input (no matter what) to exactly one of the labels in the label set. In exchange for this closed-world view, we add a layer of reliability and predictability to an LLM that already had a profound level of knowledge.
- We built an extremely simple, one-stage pipeline, but thanks to Scikit-LLM’s compatibility with scikit-learn objects and syntax, we could have created a larger pipeline with additional preprocessing steps, similar to the example in this [related article](https://machinelearningmastery.com/text-summarization-with-scikit-llm/) .
