{"slug": "serving-scikit-llm-pipelines-as-rest-apis-with-fastapi", "title": "Serving Scikit-LLM Pipelines as REST APIs with FastAPI", "summary": "A tutorial demonstrates serving a Scikit-LLM zero-shot text classification pipeline backed by a local Ollama llama3 model as an asynchronous REST API built with FastAPI, avoiding paid cloud LLM APIs. The walkthrough configures Scikit-LLM via SKLLMConfig.set_gpt_url to point at http://localhost:11434/v1/ with a dummy key, builds a ZeroShotGPTClassifier using scikit-learn's fit/predict syntax, serializes it to scikit_llm_model.joblib with joblib, and exposes it through a FastAPI endpoint tested from the interactive docs page. The stack requires Python 3.11 plus the scikit-llm, fastapi, uvicorn, joblib and pydantic packages.", "body_md": "In this article, you will learn how to serve a Scikit-LLM zero-shot text classification pipeline, powered by a local Ollama model, as a REST API with FastAPI.\n\nTopics we will cover include:\n\n- How to configure Scikit-LLM to use a local Ollama model instead of a paid cloud API.\n- How to build and serialize a zero-shot classifier using the familiar scikit-learn fit/predict syntax.\n- How to serve the classifier through an asynchronous FastAPI endpoint and test it from the interactive docs page.\n\n## Introduction\n\nIt looks like **Scikit-LLM** is here to stay. This innovative Python framework pushes the boundaries of machine learning and LLMs by treating them as two sides of the same coin, enabling scikit-learn-based pipelines fueled by powerful pre-trained LLMs adapted to perform specific tasks — for instance, building a smart zero-shot classification engine. All of this sounds enthralling until you try to put it into production and instantly get chained to costly cloud APIs, latency spikes, and so on.\n\nThis article covers the first crucial steps toward deploying a deep semantic reasoning pipeline fueled by an LLM and combined with the reliability of standardized machine learning syntax — enter the `.fit()` and `.predict()` methods — as well as predictable outputs like those found in traditional machine learning pipelines.\n\nWe will put together an open-source stack composed of **Scikit-LLM**, **FastAPI**, and **Ollama**, teaching you how to wire up a local, self-hosted LLM that acts as a deterministic text labeler: all wrapped in an asynchronous, blazing-fast REST API, thereby bypassing cloud costs and commercial lock-in.\n\n## Setup and Initial Configurations\n\nFor this practical demonstration, I have used **Visual Studio Code** with a virtual environment using Python 3.11. I have also installed some basic libraries like pandas and scikit-learn in the virtual environment. In addition, you’ll need to install the following specialized libraries to make the code work:\n\n```\npip install scikit-llm fastapi uvicorn joblib pydantic\n\n1\n\npip install scikit-llm fastapi uvicorn joblib pydantic\n```\n\nYou’ll also need to install Ollama and pull (locally download) `llama3`. This article explains the process to install Ollama in more detail, but chances are you won’t need it if you just visit the [Ollama website](https://ollama.com/download): it’s pretty simple to install, I promise!\n\n## Building, Deploying, and Testing the Pipeline\n\nOnce everything is correctly installed and you have resolved any possible issues caused by version compatibility with your Python interpreter, let’s move on to the core of the process. We’ll create a new project folder in our IDE and create two Python files inside it, which will contain the code: name them `train.py` and `app.py`.\n\nLet’s start with `train.py`. In this Python file, we will add the logic to train and serialize the Scikit-LLM pipeline. To keep things simple but scalable, we will just invoke the Ollama model and attach it to a one-stage pipeline consisting of a zero-shot classifier — in simple terms, a pre-trained LLM turned into a classifier by giving it the set of possible categories, or labels, from an example dataset of texts, without actual model training.\n\nAfter that, we serialize the pipeline: once you run the code below, a file named `scikit_llm_model.joblib` containing the serialized pipeline will be created in your project folder. Here’s the code to do this:\n\n``` python\nimport joblib\nfrom skllm.config import SKLLMConfig\nfrom skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier\n\n# 1. Routing Scikit-LLM to local Ollama instead of cloud APIs (usually paid)\nSKLLMConfig.set_gpt_url(\"http://localhost:11434/v1/\")\nSKLLMConfig.set_openai_key(\"ollama-dummy\")\n\nX = [\"The battery drains too fast\", \"I love the new camera\", \"Screen is completely cracked\"]\ny = [\"Battery Issue\", \"Positive Feedback\", \"Hardware Damage\"]\n\n# 2. Appending the specific Ollama model name after custom_url::\nclf = ZeroShotGPTClassifier(model=\"custom_url::llama3\")\nclf.fit(X, y)\n\n# 3. Serializing the estimator\njoblib.dump(clf, \"scikit_llm_model.joblib\")\nprint(\"Pipeline serialized successfully.\")\n\n123456789101112131415161718\n\nimport joblibfrom skllm.config import SKLLMConfigfrom skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier # 1. Routing Scikit-LLM to local Ollama instead of cloud APIs (usually paid)SKLLMConfig.set_gpt_url(\"http://localhost:11434/v1/\")SKLLMConfig.set_openai_key(\"ollama-dummy\") X = [\"The battery drains too fast\", \"I love the new camera\", \"Screen is completely cracked\"]y = [\"Battery Issue\", \"Positive Feedback\", \"Hardware Damage\"] # 2. Appending the specific Ollama model name after custom_url::clf = ZeroShotGPTClassifier(model=\"custom_url::llama3\")clf.fit(X, y) # 3. Serializing the estimatorjoblib.dump(clf, \"scikit_llm_model.joblib\")print(\"Pipeline serialized successfully.\")\n```\n\nThe next step is to add the code for the `app.py` file. Run this code after generating the serialized pipeline. This file is needed to:\n\n1. Reset and re-apply the Scikit-LLM configuration needed for the serving environment.\n2. Create a FastAPI app instance and define the `predict()` endpoint with the necessary logic to enable inference by the model once deployed. Note that the service can take multiple input texts in a single call, thus returning multiple class predictions in one shot.\n3. Start the server using `uvicorn` , which the final lines of code take care of.\n\n``` python\nfrom contextlib import asynccontextmanager\nimport joblib\nimport asyncio\nfrom fastapi import FastAPI, HTTPException\nfrom pydantic import BaseModel\nfrom skllm.config import SKLLMConfig\n\n# Applying configuration for the serving environment\nSKLLMConfig.set_gpt_url(\"http://localhost:11434/v1/\")\nSKLLMConfig.set_openai_key(\"ollama-dummy\")\n\nml_models = {}\n\n@asynccontextmanager\nasync def lifespan(app: FastAPI):\n    # Loading model into memory on startup\n    ml_models[\"classifier\"] = joblib.load(\"scikit_llm_model.joblib\")\n    yield\n    ml_models.clear()\n\napp = FastAPI(lifespan=lifespan)\n\nclass InferenceRequest(BaseModel):\n    texts: list[str]\n\n@app.post(\"/predict\")\nasync def predict(request: InferenceRequest):\n    clf = ml_models[\"classifier\"]\n\n    try:\n        # Offload blocking Scikit-LLM I/O to a background thread\n        predictions = await asyncio.to_thread(clf.predict, request.texts)\n        return {\"predictions\": predictions.tolist()}\n    except Exception as e:\n        raise HTTPException(status_code=500, detail=str(e))\n\nif __name__ == \"__main__\":\n    import uvicorn\n    # Starts the FastAPI server when you run 'python app.py'\n    uvicorn.run(app, host=\"127.0.0.1\", port=8000)\n\n123456789101112131415161718192021222324252627282930313233343536373839404142\n\nfrom contextlib import asynccontextmanagerimport joblibimport asynciofrom fastapi import FastAPI, HTTPExceptionfrom pydantic import BaseModelfrom skllm.config import SKLLMConfig # Applying configuration for the serving environmentSKLLMConfig.set_gpt_url(\"http://localhost:11434/v1/\")SKLLMConfig.set_openai_key(\"ollama-dummy\") ml_models = {} @asynccontextmanagerasync def lifespan(app: FastAPI):    # Loading model into memory on startup    ml_models[\"classifier\"] = joblib.load(\"scikit_llm_model.joblib\")    yield    ml_models.clear() app = FastAPI(lifespan=lifespan) class InferenceRequest(BaseModel):    texts: list[str] @app.post(\"/predict\")async def predict(request: InferenceRequest):    clf = ml_models[\"classifier\"]     try:        # Offload blocking Scikit-LLM I/O to a background thread        predictions = await asyncio.to_thread(clf.predict, request.texts)        return {\"predictions\": predictions.tolist()}    except Exception as e:        raise HTTPException(status_code=500, detail=str(e))   if __name__ == \"__main__\":    import uvicorn    # Starts the FastAPI server when you run 'python app.py'    uvicorn.run(app, host=\"127.0.0.1\", port=8000)\n```\n\nOnce you run this code in your IDE, if everything goes smoothly, you should see the following messages in the output console:\n\n```\nINFO:     Started server process [9351]\nINFO:     Waiting for application startup.\nINFO:     Application startup complete.\nINFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)\n\n1234\n\nINFO:     Started server process [9351]INFO:     Waiting for application startup.INFO:     Application startup complete.INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)\n```\n\nNow your pipeline is accessible (well, locally accessible, I mean), and you can open the FastAPI interface by browsing to [http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs). To test the app, click the “**Try it out**” button inside the expandable “**/predict**” block, and edit the “Request body” text box to paste the following input examples or any of your own:\n\n```\n{\n  \"texts\": [\n    \"My phone dies by noon even if I don't use it.\",\n    \"I dropped it on the sidewalk and the glass shattered.\",\n    \"This is the best device I have ever purchased!\"\n  ]\n}\n\n1234567\n\n{  \"texts\": [    \"My phone dies by noon even if I don't use it.\",    \"I dropped it on the sidewalk and the glass shattered.\",    \"This is the best device I have ever purchased!\"  ]}\n```\n\nThen click “Execute”, and your pipeline’s LLM will start doing its magic. In a few seconds, the predictions will pop up:\n\n```\n{\n  \"predictions\": [\n    \"Battery Issue\",\n    \"Hardware Damage\",\n    \"Positive Feedback\"\n  ]\n}\n\n1234567\n\n{  \"predictions\": [    \"Battery Issue\",    \"Hardware Damage\",    \"Positive Feedback\"  ]}\n```\n\n## Final Remarks\n\nCongratulations if you made it here! Just be aware of a few notes about what we have just done together:\n\n- The classifier we built and deployed relies on the label set `y` we passed to it in the`fit()` method early on. We didn’t update the LLM’s weights (this process would have taken considerably longer!). Instead, the classifier stored the specific closed-domain set of categories to choose from when performing inference on texts, just as if we had written a multiple-choice question in a prompt.\n- For this reason, when turning a pre-trained model into a zero-shot classifier, we are restricting its “view of the world”, forcing it to map any input (no matter what) to exactly one of the labels in the label set. In exchange for this closed-world view, we add a layer of reliability and predictability to an LLM that already had a profound level of knowledge.\n- We built an extremely simple, one-stage pipeline, but thanks to Scikit-LLM’s compatibility with scikit-learn objects and syntax, we could have created a larger pipeline with additional preprocessing steps, similar to the example in this [related article](https://machinelearningmastery.com/text-summarization-with-scikit-llm/) .", "url": "https://wpnews.pro/news/serving-scikit-llm-pipelines-as-rest-apis-with-fastapi", "canonical_source": "https://machinelearningmastery.com/serving-scikit-llm-pipelines-as-rest-apis-with-fastapi/", "published_at": "2026-11-16 11:00:00+00:00", "updated_at": "2026-11-16 17:49:26.281694+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-products"], "entities": ["Scikit-LLM", "FastAPI", "Ollama", "llama3", "ZeroShotGPTClassifier", "joblib", "Python 3.11", "Visual Studio Code"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/serving-scikit-llm-pipelines-as-rest-apis-with-fastapi", "markdown": "https://wpnews.pro/news/serving-scikit-llm-pipelines-as-rest-apis-with-fastapi.md", "text": "https://wpnews.pro/news/serving-scikit-llm-pipelines-as-rest-apis-with-fastapi.txt", "jsonld": "https://wpnews.pro/news/serving-scikit-llm-pipelines-as-rest-apis-with-fastapi.jsonld"}}