{"slug": "ai-agent-python-ollama-build-test-deploy-with-fastapi", "title": "AI agent python ollama: Build, Test, Deploy with FastAPI", "summary": "A developer built an AI agent using Ollama and FastAPI, demonstrating how to set up a local LLM server, create a Python wrapper for the Ollama API, and expose it via a FastAPI endpoint. The project uses the phi3 model and includes a calculator agent example, with deployment tips for containerization.", "body_md": "I’ll show you how to get an **ai agent python ollama** up and running on your laptop, expose it through FastAPI, test it automatically, and push it to production. You can have a working endpoint in under an hour if you follow the steps carefully.\n\nThe first thing you need is a running Ollama server. It ships as a single binary for Linux, macOS, and Windows, so there are no heavyweight dependencies.\n\n```\n# Download the latest release (replace with the version you need)\ncurl -L https://ollama.com/download/ollama-linux-amd64 -o ollama\nchmod +x ollama\nsudo mv ollama /usr/local/bin/\n```\n\nStart the daemon:\n\n```\nollama serve &\n```\n\nOllama defaults to listening on `http://127.0.0.1:11434`\n\n. You can change the bind address with the `--host`\n\nflag or set a custom port with `--port`\n\n. I keep it on the default because my FastAPI container can reach the host network via Docker’s `host`\n\nmode.\n\nNext, pull a model. Ollama supports many open-source LLMs; for a quick start I use `phi3`\n\nbecause it fits in 2 GB of RAM.\n\n```\nollama pull phi3\n```\n\nIf you’re on a low-memory box, add `--cpu`\n\nto force CPU inference. It’s slower, but it avoids OOM kills that have bitten me on cheap VPS instances.\n\n**Configuration tip:** create a `~/.ollama/config.yaml`\n\n:\n\n```\nhost: \"0.0.0.0\"\nport: 11434\nlog_level: \"info\"\n```\n\nThat way you can restart the service without remembering flags.\n\nOllama exposes a simple JSON-over-HTTP API. The core endpoint is `POST /api/chat`\n\n. Here’s a minimal wrapper that turns the HTTP call into a Python class.\n\n``` python\nimport httpx\nfrom typing import List, Dict\n\nclass OllamaChat:\n    def __init__(self, model: str = \"phi3\", base_url: str = \"http://127.0.0.1:11434\"):\n        self.base_url = base_url.rstrip(\"/\")\n        self.model = model\n        self.client = httpx.Client(timeout=30.0)\n\n    def chat(self, messages: List[Dict[str, str]]) -> str:\n        payload = {\n            \"model\": self.model,\n            \"messages\": messages,\n            \"stream\": False\n        }\n        resp = self.client.post(f\"{self.base_url}/api/chat\", json=payload)\n        resp.raise_for_status()\n        return resp.json()[\"message\"][\"content\"]\n```\n\nThe `messages`\n\nlist follows the OpenAI chat format (`role`\n\n= `system|user|assistant`\n\n). Because Ollama mimics that schema, you can reuse prompt engineering tricks you already know.\n\n**Agent pattern:** I like to encapsulate a single “goal” function. For example, a simple calculator agent:\n\n``` php\ndef calculator_agent(question: str) -> str:\n    wrapper = OllamaChat()\n    msgs = [\n        {\"role\": \"system\", \"content\": \"You are a reliable calculator. Return only the numeric answer.\"},\n        {\"role\": \"user\", \"content\": question}\n    ]\n    return wrapper.chat(msgs)\n```\n\nWhen I first built this, I forgot to set `stream: False`\n\nand got a generator back, which broke downstream code that expected a string. Always verify the response shape in a unit test.\n\nFastAPI makes wiring an endpoint trivial. Below is a complete `main.py`\n\nthat exposes `/calculate`\n\nand forwards the request to the `calculator_agent`\n\n.\n\n``` python\nfrom fastapi import FastAPI, HTTPException\nfrom pydantic import BaseModel\nfrom typing import Literal\n\napp = FastAPI(title=\"Ollama AI Agent Service\")\n\nclass CalcRequest(BaseModel):\n    expression: str\n\nclass CalcResponse(BaseModel):\n    result: str\n\n@app.post(\"/calculate\", response_model=CalcResponse)\ndef calculate(req: CalcRequest):\n    try:\n        answer = calculator_agent(req.expression)\n    except httpx.HTTPError as exc:\n        raise HTTPException(status_code=502, detail=f\"Ollama error: {exc}\") from exc\n    return CalcResponse(result=answer)\n```\n\nRun it with:\n\n```\nuvicorn main:app --host 0.0.0.0 --port 8000\n```\n\nIf you’re containerizing, keep the Ollama server in a sidecar container and share a Docker network. I once tried to run Ollama inside the same container as FastAPI, but the process died when the container restarted, leaving the API hanging. Separate containers give you independent restart policies.\n\n**Production tip:** enable `uvicorn`\n\nworkers (`--workers 4`\n\n) to utilize multiple CPU cores. Ollama itself is single-threaded per model, so you’ll want a pool of models or a load balancer if you expect high QPS.\n\nTesting LLM-driven code is tricky because the output is nondeterministic. I approach it in two layers:\n\nHere’s a pytest fixture that starts Ollama in a temporary container:\n\n``` python\nimport subprocess\nimport time\nimport pytest\n\n@pytest.fixture(scope=\"session\")\ndef ollama_server():\n    proc = subprocess.Popen([\"docker\", \"run\", \"--rm\", \"-p\", \"11434:11434\", \"ollama/ollama\"], stdout=subprocess.PIPE)\n    # Wait for the health endpoint\n    for _ in range(10):\n        try:\n            httpx.get(\"http://127.0.0.1:11434/health\").raise_for_status()\n            break\n        except Exception:\n            time.sleep(1)\n    else:\n        pytest.fail(\"Ollama did not start in time\")\n    yield\n    proc.terminate()\n```\n\nAnd a test that validates the calculator:\n\n``` python\ndef test_calculator_agent(ollama_server):\n    result = calculator_agent(\"What is 7 * 8?\")\n    assert result.strip() == \"56\"\n```\n\nIf the model ever drifts (e.g., it starts adding explanations), the test will fail, alerting you before you ship.\n\n**When NOT to rely on this:** for creative generation (storytelling, code synthesis) you can’t lock down exact output. In those cases, check for presence of required fields rather than exact string equality.\n\nRunning Ollama on a single machine works for prototypes, but production traffic often exceeds a single CPU’s capacity. My current stack uses three layers:\n\n`taskset`\n\n.\n`/api/chat`\n\nrequests round-robin to the shards.\nA minimal Kubernetes deployment for Ollama looks like:\n\n```\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: ollama\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      app: ollama\n  template:\n    metadata:\n      labels:\n        app: ollama\n    spec:\n      containers:\n        - name: ollama\n          image: ollama/ollama:latest\n          ports:\n            - containerPort: 11434\n          resources:\n            limits:\n              cpu: \"2\"\n              memory: \"4Gi\"\n          command: [\"ollama\", \"serve\", \"--host\", \"0.0.0.0\", \"--port\", \"11434\"]\n```\n\nCombine that with a Service:\n\n```\napiVersion: v1\nkind: Service\nmetadata:\n  name: ollama\nspec:\n  selector:\n    app: ollama\n  ports:\n    - protocol: TCP\n      port: 11434\n      targetPort: 11434\n```\n\nYour FastAPI pod just points to `http://ollama:11434`\n\n. If a node runs out of memory, the pod restarts – I’ve added a `livenessProbe`\n\nthat hits `/health`\n\nto catch silent OOM kills.\n\n**Cost considerations:** each model instance reserves RAM for the weights. A 3-B parameter model needs ~6 GB. Running three shards can easily hit 20 GB, so budget accordingly. If you’re on a spot-instance fleet, be prepared for occasional restarts; make your FastAPI client retry with exponential back-off.\n\n**When not to scale this way:** if you need sub-millisecond latency, Ollama’s CPU inference isn’t enough. In that case, move to a GPU-backed service or a hosted provider that offers hardware acceleration.\n\n**What language models does Ollama support?**\n\nOllama ships with many open-source models: Llama 3, Mistral, Phi-3, and community-built variants. You pull them with `ollama pull <model>`\n\nand they run on CPU or GPU depending on your hardware.\n\n**Can I use the same code with OpenAI’s API?**\n\nYes. The request schema (`model`\n\n, `messages`\n\n, `stream`\n\n) mirrors OpenAI’s chat endpoint, so swapping the base URL and API key converts the wrapper to an OpenAI client. See the related post “[Using the OpenAI Agents SDK for Python: A Practical Guide](https://www.logiclooptech.dev/using-the-openai-agents-sdk-for-python-a-practical-guide/)”.\n\n**Do I need to restart Ollama after changing the model?**\n\nNo. After pulling a new model you can start using it immediately by passing the model name to `OllamaChat`\n\n. The server loads the weights lazily on the first request.\n\n**How do I monitor Ollama’s performance?**\n\nExpose the `/metrics`\n\nendpoint (Prometheus format) by running Ollama with `--metrics`\n\n. Then scrape it with Grafana or Prometheus to watch request latency, CPU usage, and cache hits.\n\n`/api/chat`\n\nendpoint in a thin Python class; it gives you a reusable With these pieces in place, you can move from a local experiment to a production-grade AI agent service without reinventing the wheel. Happy coding!", "url": "https://wpnews.pro/news/ai-agent-python-ollama-build-test-deploy-with-fastapi", "canonical_source": "https://dev.to/ayush_kumar_085a0f2c54e3f/ai-agent-python-ollama-build-test-deploy-with-fastapi-3ho2", "published_at": "2026-07-17 09:08:29+00:00", "updated_at": "2026-07-17 09:34:25.158478+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "ai-agents"], "entities": ["Ollama", "FastAPI", "phi3", "Docker"], "alternates": {"html": "https://wpnews.pro/news/ai-agent-python-ollama-build-test-deploy-with-fastapi", "markdown": "https://wpnews.pro/news/ai-agent-python-ollama-build-test-deploy-with-fastapi.md", "text": "https://wpnews.pro/news/ai-agent-python-ollama-build-test-deploy-with-fastapi.txt", "jsonld": "https://wpnews.pro/news/ai-agent-python-ollama-build-test-deploy-with-fastapi.jsonld"}}