# AI agent python ollama: Build, Test, Deploy with FastAPI > Source: > Published: 2026-07-17 09:08:29+00:00 I’ll show you how to get an **ai agent python ollama** up and running on your laptop, expose it through FastAPI, test it automatically, and push it to production. You can have a working endpoint in under an hour if you follow the steps carefully. The first thing you need is a running Ollama server. It ships as a single binary for Linux, macOS, and Windows, so there are no heavyweight dependencies. ``` # Download the latest release (replace with the version you need) curl -L https://ollama.com/download/ollama-linux-amd64 -o ollama chmod +x ollama sudo mv ollama /usr/local/bin/ ``` Start the daemon: ``` ollama serve & ``` Ollama defaults to listening on `http://127.0.0.1:11434` . You can change the bind address with the `--host` flag or set a custom port with `--port` . I keep it on the default because my FastAPI container can reach the host network via Docker’s `host` mode. Next, pull a model. Ollama supports many open-source LLMs; for a quick start I use `phi3` because it fits in 2 GB of RAM. ``` ollama pull phi3 ``` If you’re on a low-memory box, add `--cpu` to force CPU inference. It’s slower, but it avoids OOM kills that have bitten me on cheap VPS instances. **Configuration tip:** create a `~/.ollama/config.yaml` : ``` host: "0.0.0.0" port: 11434 log_level: "info" ``` That way you can restart the service without remembering flags. Ollama exposes a simple JSON-over-HTTP API. The core endpoint is `POST /api/chat` . Here’s a minimal wrapper that turns the HTTP call into a Python class. ``` python import httpx from typing import List, Dict class OllamaChat: def __init__(self, model: str = "phi3", base_url: str = "http://127.0.0.1:11434"): self.base_url = base_url.rstrip("/") self.model = model self.client = httpx.Client(timeout=30.0) def chat(self, messages: List[Dict[str, str]]) -> str: payload = { "model": self.model, "messages": messages, "stream": False } resp = self.client.post(f"{self.base_url}/api/chat", json=payload) resp.raise_for_status() return resp.json()["message"]["content"] ``` The `messages` list follows the OpenAI chat format (`role` = `system|user|assistant` ). Because Ollama mimics that schema, you can reuse prompt engineering tricks you already know. **Agent pattern:** I like to encapsulate a single “goal” function. For example, a simple calculator agent: ``` php def calculator_agent(question: str) -> str: wrapper = OllamaChat() msgs = [ {"role": "system", "content": "You are a reliable calculator. Return only the numeric answer."}, {"role": "user", "content": question} ] return wrapper.chat(msgs) ``` When I first built this, I forgot to set `stream: False` and got a generator back, which broke downstream code that expected a string. Always verify the response shape in a unit test. FastAPI makes wiring an endpoint trivial. Below is a complete `main.py` that exposes `/calculate` and forwards the request to the `calculator_agent` . ``` python from fastapi import FastAPI, HTTPException from pydantic import BaseModel from typing import Literal app = FastAPI(title="Ollama AI Agent Service") class CalcRequest(BaseModel): expression: str class CalcResponse(BaseModel): result: str @app.post("/calculate", response_model=CalcResponse) def calculate(req: CalcRequest): try: answer = calculator_agent(req.expression) except httpx.HTTPError as exc: raise HTTPException(status_code=502, detail=f"Ollama error: {exc}") from exc return CalcResponse(result=answer) ``` Run it with: ``` uvicorn main:app --host 0.0.0.0 --port 8000 ``` If you’re containerizing, keep the Ollama server in a sidecar container and share a Docker network. I once tried to run Ollama inside the same container as FastAPI, but the process died when the container restarted, leaving the API hanging. Separate containers give you independent restart policies. **Production tip:** enable `uvicorn` workers (`--workers 4` ) to utilize multiple CPU cores. Ollama itself is single-threaded per model, so you’ll want a pool of models or a load balancer if you expect high QPS. Testing LLM-driven code is tricky because the output is nondeterministic. I approach it in two layers: Here’s a pytest fixture that starts Ollama in a temporary container: ``` python import subprocess import time import pytest @pytest.fixture(scope="session") def ollama_server(): proc = subprocess.Popen(["docker", "run", "--rm", "-p", "11434:11434", "ollama/ollama"], stdout=subprocess.PIPE) # Wait for the health endpoint for _ in range(10): try: httpx.get("http://127.0.0.1:11434/health").raise_for_status() break except Exception: time.sleep(1) else: pytest.fail("Ollama did not start in time") yield proc.terminate() ``` And a test that validates the calculator: ``` python def test_calculator_agent(ollama_server): result = calculator_agent("What is 7 * 8?") assert result.strip() == "56" ``` If the model ever drifts (e.g., it starts adding explanations), the test will fail, alerting you before you ship. **When NOT to rely on this:** for creative generation (storytelling, code synthesis) you can’t lock down exact output. In those cases, check for presence of required fields rather than exact string equality. Running Ollama on a single machine works for prototypes, but production traffic often exceeds a single CPU’s capacity. My current stack uses three layers: `taskset` . `/api/chat` requests round-robin to the shards. A minimal Kubernetes deployment for Ollama looks like: ``` apiVersion: apps/v1 kind: Deployment metadata: name: ollama spec: replicas: 3 selector: matchLabels: app: ollama template: metadata: labels: app: ollama spec: containers: - name: ollama image: ollama/ollama:latest ports: - containerPort: 11434 resources: limits: cpu: "2" memory: "4Gi" command: ["ollama", "serve", "--host", "0.0.0.0", "--port", "11434"] ``` Combine that with a Service: ``` apiVersion: v1 kind: Service metadata: name: ollama spec: selector: app: ollama ports: - protocol: TCP port: 11434 targetPort: 11434 ``` Your FastAPI pod just points to `http://ollama:11434` . If a node runs out of memory, the pod restarts – I’ve added a `livenessProbe` that hits `/health` to catch silent OOM kills. **Cost considerations:** each model instance reserves RAM for the weights. A 3-B parameter model needs ~6 GB. Running three shards can easily hit 20 GB, so budget accordingly. If you’re on a spot-instance fleet, be prepared for occasional restarts; make your FastAPI client retry with exponential back-off. **When not to scale this way:** if you need sub-millisecond latency, Ollama’s CPU inference isn’t enough. In that case, move to a GPU-backed service or a hosted provider that offers hardware acceleration. **What language models does Ollama support?** Ollama ships with many open-source models: Llama 3, Mistral, Phi-3, and community-built variants. You pull them with `ollama pull ` and they run on CPU or GPU depending on your hardware. **Can I use the same code with OpenAI’s API?** Yes. The request schema (`model` , `messages` , `stream` ) mirrors OpenAI’s chat endpoint, so swapping the base URL and API key converts the wrapper to an OpenAI client. See the related post “[Using the OpenAI Agents SDK for Python: A Practical Guide](https://www.logiclooptech.dev/using-the-openai-agents-sdk-for-python-a-practical-guide/)”. **Do I need to restart Ollama after changing the model?** No. After pulling a new model you can start using it immediately by passing the model name to `OllamaChat` . The server loads the weights lazily on the first request. **How do I monitor Ollama’s performance?** Expose the `/metrics` endpoint (Prometheus format) by running Ollama with `--metrics` . Then scrape it with Grafana or Prometheus to watch request latency, CPU usage, and cache hits. `/api/chat` endpoint in a thin Python class; it gives you a reusable With these pieces in place, you can move from a local experiment to a production-grade AI agent service without reinventing the wheel. Happy coding!