AI agent python ollama: Build, Test, Deploy with FastAPI A developer built an AI agent using Ollama and FastAPI, demonstrating how to set up a local LLM server, create a Python wrapper for the Ollama API, and expose it via a FastAPI endpoint. The project uses the phi3 model and includes a calculator agent example, with deployment tips for containerization. I’ll show you how to get an ai agent python ollama up and running on your laptop, expose it through FastAPI, test it automatically, and push it to production. You can have a working endpoint in under an hour if you follow the steps carefully. The first thing you need is a running Ollama server. It ships as a single binary for Linux, macOS, and Windows, so there are no heavyweight dependencies. Download the latest release replace with the version you need curl -L https://ollama.com/download/ollama-linux-amd64 -o ollama chmod +x ollama sudo mv ollama /usr/local/bin/ Start the daemon: ollama serve & Ollama defaults to listening on http://127.0.0.1:11434 . You can change the bind address with the --host flag or set a custom port with --port . I keep it on the default because my FastAPI container can reach the host network via Docker’s host mode. Next, pull a model. Ollama supports many open-source LLMs; for a quick start I use phi3 because it fits in 2 GB of RAM. ollama pull phi3 If you’re on a low-memory box, add --cpu to force CPU inference. It’s slower, but it avoids OOM kills that have bitten me on cheap VPS instances. Configuration tip: create a ~/.ollama/config.yaml : host: "0.0.0.0" port: 11434 log level: "info" That way you can restart the service without remembering flags. Ollama exposes a simple JSON-over-HTTP API. The core endpoint is POST /api/chat . Here’s a minimal wrapper that turns the HTTP call into a Python class. python import httpx from typing import List, Dict class OllamaChat: def init self, model: str = "phi3", base url: str = "http://127.0.0.1:11434" : self.base url = base url.rstrip "/" self.model = model self.client = httpx.Client timeout=30.0 def chat self, messages: List Dict str, str - str: payload = { "model": self.model, "messages": messages, "stream": False } resp = self.client.post f"{self.base url}/api/chat", json=payload resp.raise for status return resp.json "message" "content" The messages list follows the OpenAI chat format role = system|user|assistant . Because Ollama mimics that schema, you can reuse prompt engineering tricks you already know. Agent pattern: I like to encapsulate a single “goal” function. For example, a simple calculator agent: php def calculator agent question: str - str: wrapper = OllamaChat msgs = {"role": "system", "content": "You are a reliable calculator. Return only the numeric answer."}, {"role": "user", "content": question} return wrapper.chat msgs When I first built this, I forgot to set stream: False and got a generator back, which broke downstream code that expected a string. Always verify the response shape in a unit test. FastAPI makes wiring an endpoint trivial. Below is a complete main.py that exposes /calculate and forwards the request to the calculator agent . python from fastapi import FastAPI, HTTPException from pydantic import BaseModel from typing import Literal app = FastAPI title="Ollama AI Agent Service" class CalcRequest BaseModel : expression: str class CalcResponse BaseModel : result: str @app.post "/calculate", response model=CalcResponse def calculate req: CalcRequest : try: answer = calculator agent req.expression except httpx.HTTPError as exc: raise HTTPException status code=502, detail=f"Ollama error: {exc}" from exc return CalcResponse result=answer Run it with: uvicorn main:app --host 0.0.0.0 --port 8000 If you’re containerizing, keep the Ollama server in a sidecar container and share a Docker network. I once tried to run Ollama inside the same container as FastAPI, but the process died when the container restarted, leaving the API hanging. Separate containers give you independent restart policies. Production tip: enable uvicorn workers --workers 4 to utilize multiple CPU cores. Ollama itself is single-threaded per model, so you’ll want a pool of models or a load balancer if you expect high QPS. Testing LLM-driven code is tricky because the output is nondeterministic. I approach it in two layers: Here’s a pytest fixture that starts Ollama in a temporary container: python import subprocess import time import pytest @pytest.fixture scope="session" def ollama server : proc = subprocess.Popen "docker", "run", "--rm", "-p", "11434:11434", "ollama/ollama" , stdout=subprocess.PIPE Wait for the health endpoint for in range 10 : try: httpx.get "http://127.0.0.1:11434/health" .raise for status break except Exception: time.sleep 1 else: pytest.fail "Ollama did not start in time" yield proc.terminate And a test that validates the calculator: python def test calculator agent ollama server : result = calculator agent "What is 7 8?" assert result.strip == "56" If the model ever drifts e.g., it starts adding explanations , the test will fail, alerting you before you ship. When NOT to rely on this: for creative generation storytelling, code synthesis you can’t lock down exact output. In those cases, check for presence of required fields rather than exact string equality. Running Ollama on a single machine works for prototypes, but production traffic often exceeds a single CPU’s capacity. My current stack uses three layers: taskset . /api/chat requests round-robin to the shards. A minimal Kubernetes deployment for Ollama looks like: apiVersion: apps/v1 kind: Deployment metadata: name: ollama spec: replicas: 3 selector: matchLabels: app: ollama template: metadata: labels: app: ollama spec: containers: - name: ollama image: ollama/ollama:latest ports: - containerPort: 11434 resources: limits: cpu: "2" memory: "4Gi" command: "ollama", "serve", "--host", "0.0.0.0", "--port", "11434" Combine that with a Service: apiVersion: v1 kind: Service metadata: name: ollama spec: selector: app: ollama ports: - protocol: TCP port: 11434 targetPort: 11434 Your FastAPI pod just points to http://ollama:11434 . If a node runs out of memory, the pod restarts – I’ve added a livenessProbe that hits /health to catch silent OOM kills. Cost considerations: each model instance reserves RAM for the weights. A 3-B parameter model needs ~6 GB. Running three shards can easily hit 20 GB, so budget accordingly. If you’re on a spot-instance fleet, be prepared for occasional restarts; make your FastAPI client retry with exponential back-off. When not to scale this way: if you need sub-millisecond latency, Ollama’s CPU inference isn’t enough. In that case, move to a GPU-backed service or a hosted provider that offers hardware acceleration. What language models does Ollama support? Ollama ships with many open-source models: Llama 3, Mistral, Phi-3, and community-built variants. You pull them with ollama pull