I’ll show you how to get an ai agent python ollama up and running on your laptop, expose it through FastAPI, test it automatically, and push it to production. You can have a working endpoint in under an hour if you follow the steps carefully.
The first thing you need is a running Ollama server. It ships as a single binary for Linux, macOS, and Windows, so there are no heavyweight dependencies.
curl -L https://ollama.com/download/ollama-linux-amd64 -o ollama
chmod +x ollama
sudo mv ollama /usr/local/bin/
Start the daemon:
ollama serve &
Ollama defaults to listening on http://127.0.0.1:11434
. You can change the bind address with the --host
flag or set a custom port with --port
. I keep it on the default because my FastAPI container can reach the host network via Docker’s host
mode.
Next, pull a model. Ollama supports many open-source LLMs; for a quick start I use phi3
because it fits in 2 GB of RAM.
ollama pull phi3
If you’re on a low-memory box, add --cpu
to force CPU inference. It’s slower, but it avoids OOM kills that have bitten me on cheap VPS instances.
Configuration tip: create a ~/.ollama/config.yaml
:
host: "0.0.0.0"
port: 11434
log_level: "info"
That way you can restart the service without remembering flags.
Ollama exposes a simple JSON-over-HTTP API. The core endpoint is POST /api/chat
. Here’s a minimal wrapper that turns the HTTP call into a Python class.
import httpx
from typing import List, Dict
class OllamaChat:
def __init__(self, model: str = "phi3", base_url: str = "http://127.0.0.1:11434"):
self.base_url = base_url.rstrip("/")
self.model = model
self.client = httpx.Client(timeout=30.0)
def chat(self, messages: List[Dict[str, str]]) -> str:
payload = {
"model": self.model,
"messages": messages,
"stream": False
}
resp = self.client.post(f"{self.base_url}/api/chat", json=payload)
resp.raise_for_status()
return resp.json()["message"]["content"]
The messages
list follows the OpenAI chat format (role
= system|user|assistant
). Because Ollama mimics that schema, you can reuse prompt engineering tricks you already know.
Agent pattern: I like to encapsulate a single “goal” function. For example, a simple calculator agent:
def calculator_agent(question: str) -> str:
wrapper = OllamaChat()
msgs = [
{"role": "system", "content": "You are a reliable calculator. Return only the numeric answer."},
{"role": "user", "content": question}
]
return wrapper.chat(msgs)
When I first built this, I forgot to set stream: False
and got a generator back, which broke downstream code that expected a string. Always verify the response shape in a unit test.
FastAPI makes wiring an endpoint trivial. Below is a complete main.py
that exposes /calculate
and forwards the request to the calculator_agent
.
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import Literal
app = FastAPI(title="Ollama AI Agent Service")
class CalcRequest(BaseModel):
expression: str
class CalcResponse(BaseModel):
result: str
@app.post("/calculate", response_model=CalcResponse)
def calculate(req: CalcRequest):
try:
answer = calculator_agent(req.expression)
except httpx.HTTPError as exc:
raise HTTPException(status_code=502, detail=f"Ollama error: {exc}") from exc
return CalcResponse(result=answer)
Run it with:
uvicorn main:app --host 0.0.0.0 --port 8000
If you’re containerizing, keep the Ollama server in a sidecar container and share a Docker network. I once tried to run Ollama inside the same container as FastAPI, but the process died when the container restarted, leaving the API hanging. Separate containers give you independent restart policies.
Production tip: enable uvicorn
workers (--workers 4
) to utilize multiple CPU cores. Ollama itself is single-threaded per model, so you’ll want a pool of models or a load balancer if you expect high QPS.
Testing LLM-driven code is tricky because the output is nondeterministic. I approach it in two layers:
Here’s a pytest fixture that starts Ollama in a temporary container:
import subprocess
import time
import pytest
@pytest.fixture(scope="session")
def ollama_server():
proc = subprocess.Popen(["docker", "run", "--rm", "-p", "11434:11434", "ollama/ollama"], stdout=subprocess.PIPE)
for _ in range(10):
try:
httpx.get("http://127.0.0.1:11434/health").raise_for_status()
break
except Exception:
time.sleep(1)
else:
pytest.fail("Ollama did not start in time")
yield
proc.terminate()
And a test that validates the calculator:
def test_calculator_agent(ollama_server):
result = calculator_agent("What is 7 * 8?")
assert result.strip() == "56"
If the model ever drifts (e.g., it starts adding explanations), the test will fail, alerting you before you ship.
When NOT to rely on this: for creative generation (storytelling, code synthesis) you can’t lock down exact output. In those cases, check for presence of required fields rather than exact string equality.
Running Ollama on a single machine works for prototypes, but production traffic often exceeds a single CPU’s capacity. My current stack uses three layers:
taskset
.
/api/chat
requests round-robin to the shards. A minimal Kubernetes deployment for Ollama looks like:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ollama
spec:
replicas: 3
selector:
matchLabels:
app: ollama
template:
metadata:
labels:
app: ollama
spec:
containers:
- name: ollama
image: ollama/ollama:latest
ports:
- containerPort: 11434
resources:
limits:
cpu: "2"
memory: "4Gi"
command: ["ollama", "serve", "--host", "0.0.0.0", "--port", "11434"]
Combine that with a Service:
apiVersion: v1
kind: Service
metadata:
name: ollama
spec:
selector:
app: ollama
ports:
- protocol: TCP
port: 11434
targetPort: 11434
Your FastAPI pod just points to http://ollama:11434
. If a node runs out of memory, the pod restarts – I’ve added a livenessProbe
that hits /health
to catch silent OOM kills.
Cost considerations: each model instance reserves RAM for the weights. A 3-B parameter model needs ~6 GB. Running three shards can easily hit 20 GB, so budget accordingly. If you’re on a spot-instance fleet, be prepared for occasional restarts; make your FastAPI client retry with exponential back-off.
When not to scale this way: if you need sub-millisecond latency, Ollama’s CPU inference isn’t enough. In that case, move to a GPU-backed service or a hosted provider that offers hardware acceleration.
What language models does Ollama support?
Ollama ships with many open-source models: Llama 3, Mistral, Phi-3, and community-built variants. You pull them with ollama pull <model>
and they run on CPU or GPU depending on your hardware.
Can I use the same code with OpenAI’s API?
Yes. The request schema (model
, messages
, stream
) mirrors OpenAI’s chat endpoint, so swapping the base URL and API key converts the wrapper to an OpenAI client. See the related post “Using the OpenAI Agents SDK for Python: A Practical Guide”.
Do I need to restart Ollama after changing the model?
No. After pulling a new model you can start using it immediately by passing the model name to OllamaChat
. The server loads the weights lazily on the first request.
How do I monitor Ollama’s performance?
Expose the /metrics
endpoint (Prometheus format) by running Ollama with --metrics
. Then scrape it with Grafana or Prometheus to watch request latency, CPU usage, and cache hits.
/api/chat
endpoint in a thin Python class; it gives you a reusable With these pieces in place, you can move from a local experiment to a production-grade AI agent service without reinventing the wheel. Happy coding!