Ollama API Tutorial: 12 Steps, 70 Min A tutorial published by Shattered walks developers through 12 working calls against the Ollama API in about 70 minutes, building a Python project called `ollama-api-lab` that chats, streams tokens, extracts structured JSON, calls tools and searches documents with embeddings. The tutorial notes that Ollama serves all local models on localhost:11434 with no API key, exposes native `/api` endpoints alongside OpenAI-compatible `/v1` and Anthropic-compatible `/v1/messages` layers, and states the API is not strictly versioned but is expected to remain stable and backward compatible. Ollama's local FAQ states "We don't see your prompts or data when you run locally," though the tutorial warns that prompts sent to a model tagged `:cloud` do leave the machine. Every model you pull into Ollama becomes an HTTP service the moment the server starts. The LlamaIndex integration docs put it plainly: “All of your local models are automatically served on localhost:11434” https://developers.llamaindex.ai/python/framework/integrations/llm/ollama/ . That one fact turns a chat toy into infrastructure, because any script, editor plugin or backend job on the machine can call the Ollama API with nothing more than an HTTP client. Most tutorials stop at ollama run and a chat prompt. The harder questions come later: how to stream tokens without buffering, how to force valid JSON, how to let a model call your functions, how to keep memory in check and how to stop a laptop server from leaking onto the office network. If you have already followed a llama.cpp route, such as running Qwen3.8-27B through llama.cpp https://shattered.io/run-llm-locally-qwen3-8-27b-llama-cpp-12-steps-2026/ , you know how much work a raw runtime leaves to you. Ollama trades some of that control for a stable REST surface, and this tutorial uses that surface end to end. You will make 12 working API calls in about 70 minutes, then wire them into one small Python project called ollama-api-lab that chats, streams, extracts structured data, calls tools and searches documents with embeddings. Every command was chosen to run on a plain Linux, macOS or Windows machine with Python and curl installed. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F What the Ollama API Covers in November 2026 Ollama exposes three request styles on one port. The native API lives under /api and carries every feature first. An OpenAI-compatible layer lives under /v1 , so existing OpenAI client code can point at a local model by changing one base URL. Ollama’s documentation also lists an Anthropic-compatible mode, where the Anthropic client appends /v1/messages to the base URL you give it. All three run on the same local server, and none of the local calls need an API key. The native endpoints, as listed in Ollama’s API reference, are /api/generate , /api/chat , /api/embed , /api/tags , /api/ps , /api/show , /api/create , /api/copy , /api/pull , /api/push , /api/delete and /api/version . Two further entries, /api/usage and /api/balance , appear in the same list. Ollama states that its API is not strictly versioned but is expected to stay stable and backward compatible, with rare deprecations announced in the release notes. For a developer, that promise matters more than any single feature: code you write today against /api/chat should keep working through upgrades. The product around the API has moved fast this autumn. Ollama added support for speech models, and its documentation requires Ollama v0.40.9 or later for Phonon. On November 16, 2026, it described prefix-caching work aimed at long coding-agent conversations. It also changed its paid cloud plans so that Pro, Max and Team use per-token pricing with usage included in every plan. According to the TokenPlans tracker, the November 13 pricing table https://tokenplans.dev/changelog/2026-11-13-ollama--models/ kept Pro, Max and Team at $20, $100 and $500 a month. None of that affects local use. Ollama’s documentation does not say local inference requires a paid plan, and cloud models are a separate product reached through the same app, CLI and API. Privacy is the other reason developers choose the local path. Ollama’s local FAQ states: “We don’t see your prompts or data when you run locally.” If you call a model tagged :cloud , your prompt does leave the machine, so keep that distinction clear whenever you review a script. Every example in this tutorial uses a model that runs on your own hardware. For a wider view of how the Ollama runtime stacks up against its rivals, see the comparison of Ollama, LM Studio, llama.cpp and vLLM https://shattered.io/ollama-vs-lm-studio-vs-llama-cpp-vs-vllm-local-llm-runtimes-2026/ , and browse the AI and machine learning section https://shattered.io/category/ai-machine-learning/ for the model news that feeds your choices here. Which base URL should you call? The table below lists the base URLs from Ollama’s API introduction. Pick the row that matches your client, then keep it in one environment variable so you can swap targets without editing code. | Target | Base URL | Authentication | |---|---|---| | Local native API | http://localhost:11434/api | None | | Local OpenAI-compatible | http://localhost:11434/v1 | Placeholder key only | | Local Anthropic-style client | http://localhost:11434 client appends /v1/messages | None | | Hosted native API | https://ollama.com/api | Bearer API key | | Hosted OpenAI-compatible | https://ollama.com/v1 | Bearer API key | Creating and deleting models works only against a local server. That detail trips up people who test against the hosted API first and then wonder why /api/create fails. Prerequisites and Versions You need a machine that can hold one small model in memory, a terminal and about 70 minutes. The version floors below are the ones this tutorial was written against. Newer releases work too. | Component | Version | Why you need it | |---|---|---| | Ollama | 0.40.9 or later | The floor Ollama’s docs set for its newest model types, so it carries the current API surface | | Python | 3.10 or later | Runs the project files | | requests | Not listed | Plain HTTP client, used for every raw API call | | openai optional | 1.x | Step 11 compatibility test | | curl | Not listed | First-call examples | | jq | 1.6 or later | Readable JSON output in the terminal | | Free disk | Size shown on the model’s library page | Weights are downloaded once and cached | Hardware depends on the model you choose, and Ollama’s library page for each tag lists its download size. Check it before you pull. Small tags suit a laptop, while the larger tags want a workstation GPU or a lot of unified memory. If you own a flagship card, the RTX 5090 local LLM walkthrough https://shattered.io/run-llm-locally-rtx-5090/ shows what that class of hardware handles. Ollama’s FAQ explains how to tell where a model landed: run ollama ps , and the PROCESSOR column reports 100% GPU , 100% CPU or a split such as 48%/52% CPU/GPU when the model spills across both. Create the project folder and a virtual environment now so the later steps stay clean. mkdir ollama-api-lab && cd ollama-api-lab python3 -m venv .venv source .venv/bin/activate Windows: .venv\Scripts\activate pip install "requests =2.31" openai mkdir -p data If the install finishes without errors, run python -c "import requests; print requests. version " and confirm it prints a version number instead of an import error. A step-by-step walkthrough such as this Founderz guide to running a model with Ollama https://founderz.com/blog/ollama/ covers the installer screens if you want screenshots of the desktop app. Steps 1 to 3: Install, Pull a Model, Make the First Call Step 1: Install Ollama and confirm the server answers On macOS and Windows, download the desktop installer from ollama.com and launch it. On Linux, use the installer script from Ollama’s documentation. curl -fsSL https://ollama.com/install.sh | sh Ollama’s FAQ says the same script upgrades an existing Linux install, while macOS and Windows apps download updates automatically and offer a “Restart to update” option in the menu bar or taskbar. On Linux the installer registers a systemd service. On the desktop apps, the server starts with the app. If neither is running, start it by hand with ollama serve . Now prove the server is alive and check its version. The /api/version endpoint needs no model. curl -s http://localhost:11434/api/version ollama --version Sample output your version string will differ : {"version":"0.40.9"} If the JSON appears, port 11434 is open and the HTTP layer works. If curl reports “connection refused,” the server is not running. Fix that before going further, because every later step depends on it. Also compare the number to the 0.40.9 floor from the prerequisites table. A lower value means you should rerun the installer. Step 2: Pull a model and pick one for API work Ollama’s own documentation shows ollama run gemma4 as its example, including an image question, and it also lists a larger gemma4:31b tag. For this tutorial, use the default gemma4 tag as the chat model and embeddinggemma for embeddings. Ollama’s CLI reference uses both names in its examples. Pull them now, because the Ollama documentation states the rule directly: for local inference, you download a model before you use it. ollama pull gemma4 ollama pull embeddinggemma ollama ls Sample ollama ls output IDs and sizes are placeholders, yours will differ : NAME ID SIZE MODIFIED embeddinggemma:latest a1b2c3d4e5f6 x.x GB 10 seconds ago gemma4:latest f6e5d4c3b2a1 x.x GB 2 minutes ago The table below compares the models mentioned in this tutorial. All rows come from Ollama’s pages or NVIDIA’s own model documentation. | Model | Role here | What the official sources say | |---|---|---| | gemma4 | Chat, vision, tools | Used in Ollama’s CLI and API examples, with a larger gemma4:31b tag | | embeddinggemma | Embeddings Step 9 | Used in Ollama’s CLI reference for embedding examples | | nomic-embed-text | Alternative embeddings | Also shown in Ollama’s CLI reference | | NVIDIA Nemotron 3.5 Bolt E4B 4GB | Newest small model to try | 8B total and 4B effective parameters, a 64K context window and 35 languages | | NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB | Large local agent model | Released October 15, 2026, uses native 2-bit quantization to fit 24 GB | Treat the Nemotron rows as an upgrade path. NVIDIA’s documentation lists a Q4 K M GGUF checkpoint named NVIDIA-Nemotron-3.5-Bolt-E4B-4GB and states that the model supports Ollama, llama.cpp, vLLM, Unsloth and LM Studio. Ollama’s library reportedly carries nemotron-3.5-bolt:e4b-mxfp8 and nemotron-3.5-bolt:e4b-mlx-bf16 entries, but those tag names are single-source, so open the library page and confirm the exact tag before you pull it. You can read the model details in NVIDIA’s arXiv paper on Nemotron 3.5 Bolt E4B 4GB https://arxiv.org/abs/2611.20771 and in the NeMo AutoModel page for the BF16 checkpoint https://docs.nvidia.com/nemo/automodel/nightly/model-coverage/vision-language-models/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-BF16 . If your goal is a code assistant rather than a general API, weigh the options in the best local LLM for coding ranking https://shattered.io/best-local-llm-for-coding-free-weights-vs-2-api-2026/ before you commit to one model. A coding-tuned model changes which tools and context sizes make sense in Steps 6 and 8. Keep the model name in one environment variable. Every later script reads it, so swapping models later takes a single line. export OLLAMA MODEL=gemma4 export OLLAMA EMBED MODEL=embeddinggemma Step 3: Send your first /api/generate request The /api/generate endpoint takes one prompt and returns one completion. It is the simplest call in the API and a good smoke test. Set "stream": false to get one JSON object back instead of a stream of fragments. The endpoint streams by default, which surprises many first-time users. curl -s http://localhost:11434/api/generate -d '{ "model": "gemma4", "prompt": "In one sentence, what is a context window?", "stream": false }' | jq '{response, done, done reason, eval count, total duration}' Sample output text and timings vary on every run : { "response": "A context window is the maximum amount of text, measured in tokens, that a model can consider at once when producing its next output.", "done": true, "done reason": "stop", "eval count": 31, "total duration": 2841637000 } Four fields deserve attention. response holds the text. done reason tells you why generation ended, and stop means the model finished on its own, while length means it hit a token cap. eval count counts generated tokens. total duration is in nanoseconds and includes model load time, so the first call after a cold start looks slow and later calls look fast. Divide eval count by eval duration also in nanoseconds and multiply by 1,000,000,000 to get tokens per second, the one speed figure worth tracking. You can also pass "system" to set behavior and "options" to change sampling. Both appear again in Step 6. If your machine is a DGX-class box rather than a laptop, the DGX Spark Ollama and vLLM tutorial https://shattered.io/run-llm-dgx-spark-ollama-vllm-12-steps-2026/ shows how the same calls behave on bigger memory. Steps 4 to 6: Chat, Streaming and Generation Controls Step 4: Chat with /api/chat and keep the history yourself The /api/chat endpoint takes a list of messages, each with a role of system , user , assistant or tool . The server keeps no conversation state between requests. If you want a model to remember that your name is Dana, you must send the earlier turns again with every call. Beginners often expect the opposite and read a forgetful model as a bug. curl -s http://localhost:11434/api/chat -d '{ "model": "gemma4", "messages": {"role": "system", "content": "You answer in at most two sentences."}, {"role": "user", "content": "Why is the sky blue?"} , "stream": false }' | jq '.message' Sample output wording will vary : { "role": "assistant", "content": "Sunlight scatters off air molecules, and shorter blue wavelengths scatter far more than red ones. Our eyes catch that scattered blue light from every direction." } Ollama’s API introduction tells you to read the answer from message.content . The same shape works for local and hosted models, which makes it the best default for application code. Now create the wrapper used by every later step. Save this as client.py in the project folder. It holds one session, turns HTTP failures into a readable OllamaError , and exposes a method for each endpoint family you will touch. python import json import os import requests BASE = os.environ.get "OLLAMA BASE", "http://localhost:11434" MODEL = os.environ.get "OLLAMA MODEL", "gemma4" EMBED MODEL = os.environ.get "OLLAMA EMBED MODEL", "embeddinggemma" class OllamaError RuntimeError : pass class Ollama: def init self, base=BASE, model=MODEL, timeout=300 : self.base = base.rstrip "/" self.model = model self.timeout = timeout self.http = requests.Session def request self, method, path, payload=None, stream=False : try: resp = self.http.request method, self.base + path, json=payload, stream=stream, timeout=self.timeout, except requests.ConnectionError as exc: raise OllamaError f"cannot reach {self.base}, is 'ollama serve' running?" from exc if resp.status code = 200: try: detail = resp.json .get "error", resp.text except ValueError: detail = resp.text raise OllamaError f"HTTP {resp.status code} on {path}: {detail}" return resp def version self : return self. request "GET", "/api/version" .json "version" def models self : return m "name" for m in self. request "GET", "/api/tags" .json "models" def running self : return self. request "GET", "/api/ps" .json "models" def generate self, prompt, extra : body = {"model": self.model, "prompt": prompt, "stream": False, extra} return self. request "POST", "/api/generate", body .json def chat self, messages, extra : body = {"model": self.model, "messages": messages, "stream": False, extra} return self. request "POST", "/api/chat", body .json def stream chat self, messages, extra : body = {"model": self.model, "messages": messages, "stream": True, extra} with self. request "POST", "/api/chat", body, stream=True as resp: for line in resp.iter lines : if not line: continue chunk = json.loads line if "error" in chunk: raise OllamaError chunk "error" yield chunk def embed self, texts, model=EMBED MODEL : body = {"model": model, "input": texts} return self. request "POST", "/api/embed", body .json "embeddings" def unload self : self. request "POST", "/api/generate", {"model": self.model, "keep alive": 0} def tokens per second reply : seconds = reply "eval duration" / 1e9 return reply "eval count" / seconds if seconds else 0.0 Next, prove the memory behavior with a short script, chat demo.py . It appends each assistant reply to the history before the next question, which is the whole trick. python from client import Ollama llm = Ollama history = {"role": "system", "content": "You are a terse assistant."} for question in "My name is Dana.", "What name did I just give you?" : history.append {"role": "user", "content": question} reply = llm.chat history history.append reply "message" print question, "- ", reply "message" "content" Run it with python chat demo.py . The second answer should contain “Dana”. For long sessions, remember the context limit: Ollama’s FAQ says the default context window is 4096 tokens, so a history that grows past that gets truncated from the front. Trim old turns or raise the limit in Step 6. Step 5: Stream tokens as they arrive Streaming is the default for both /api/generate and /api/chat , and it is the right choice for anything a person reads. The server sends newline-delimited JSON, one object per line, and the last object carries "done": true plus the timing stats. Use curl with the -N flag so it does not buffer output. curl -sN http://localhost:11434/api/chat -d '{ "model": "gemma4", "messages": {"role": "user", "content": "Name three uses of DNS caching."} }' Each streamed line holds a message.content fragment and a done flag, and the final line adds done reason and eval count . Each line is a complete JSON document, so you can parse lines independently. Do not wait for the full body and then call json.loads on all of it, because that fails on the first newline. The stream chat method in client.py already reads one line at a time and raises if the server sends an {"error": ...} object mid-stream, which happens when a model crashes or runs out of memory after the HTTP 200 header has already gone out. Save the next file as stream demo.py . python import sys import time from client import Ollama llm = Ollama messages = {"role": "user", "content": "Explain DNS caching in three sentences."} start = time.perf counter first = None for chunk in llm.stream chat messages : piece = chunk "message" "content" if piece and first is None: first = time.perf counter - start sys.stdout.write piece sys.stdout.flush if chunk.get "done" : print f"\n\n first token {first:.2f}s, {chunk 'eval count' } tokens, {chunk 'done reason' } " Time to first token is the number users feel. It includes model loading on a cold start, so run the script twice and compare. Reasoning-style models may also emit a separate thinking field in each chunk’s message. If you print only content , you will see a pause before text appears while the model reasons. Decide whether your app should show, hide or log that field. Step 6: Control generation with options, context length and keep alive Sampling and memory settings go in the options object, and model residency goes in keep alive . Ollama’s FAQ shows the pattern for context size: pass num ctx inside options . It also documents the OLLAMA CONTEXT LENGTH environment variable for changing the server-wide default and /set parameter num ctx inside an interactive session. | Option | What it controls | A practical setting | |---|---|---| | temperature | Randomness of token choice | 0 for extraction and tests, 0.7 for chat | | num ctx | Context window in tokens default 4096 per the FAQ | Raise it only as far as your memory allows | | num predict | Maximum tokens to generate | Set a cap so a runaway answer cannot stall a request | | seed | Random seed | Fix it with temperature 0 for repeatable tests | | stop | Strings that end generation | Use it to cut output at a delimiter | | keep alive top level | How long the model stays in memory | "10m" for a busy service, 0 to unload now | Larger num ctx values cost memory, because the key-value cache grows with the window, and a model that no longer fits entirely in GPU memory slows down sharply as layers move to the CPU. Raise the window in steps and watch ollama ps . curl -s http://localhost:11434/api/generate -d '{ "model": "gemma4", "prompt": "List five HTTP status codes as a plain list.", "stream": false, "keep alive": "10m", "options": {"temperature": 0, "num ctx": 8192, "num predict": 120, "seed": 7} }' | jq -r '.response' curl -s http://localhost:11434/api/ps | jq '.models | {name, size, size vram, expires at}' The second call lists loaded models with a size vram value that shows how much of the model sits in GPU memory. Compare size with size vram . If size vram is smaller, part of the model is running on the CPU. To free memory right away, send a request with keep alive set to 0, which is exactly what Ollama.unload does. curl -s http://localhost:11434/api/generate -d '{"model": "gemma4", "keep alive": 0}' Sample output of that call: {"model":"gemma4","created at":"2026-11-19T10:44:51Z","response":"","done":true,"done reason":"unload"} A reply of done reason: unload confirms the model left memory. By default, Ollama keeps a model loaded for five minutes after the last request, which is why a second call a minute later feels instant while one after lunch feels slow. Steps 7 to 9: Structured Output, Tool Calling and Embeddings Step 7: Force valid JSON with a schema Free-form text is a poor interface for software. Ollama’s documentation lists Structured Outputs among its capabilities: pass a JSON schema in the format field, and the server constrains generation so the reply matches it. Compared with begging the model to “return JSON only,” schema-constrained output removes the stray prose and the markdown fences that break parsers. Create a sample document first. cat data/invoice.txt <<'EOF' Invoice 2291 from Northwind Print Co. Date due: 2026-12-05 Business cards x500 ..... 84.00 USD Poster printing x12 ..... 216.50 USD Rush fee ................. 40.00 USD Total due: 340.50 USD EOF Then save this as extract.py . The schema marks vendor , total , currency and line items as required, and the code double-checks them anyway. Validation after generation is cheap and catches the rare case where a model returns an empty object. python import json import sys from client import Ollama SCHEMA = { "type": "object", "properties": { "vendor": {"type": "string"}, "total": {"type": "number"}, "currency": {"type": "string"}, "due date": {"type": "string"}, "line items": { "type": "array", "items": { "type": "object", "properties": { "description": {"type": "string"}, "amount": {"type": "number"}, }, "required": "description", "amount" , }, }, }, "required": "vendor", "total", "currency", "line items" , } def extract invoice text, llm=None : llm = llm or Ollama reply = llm.chat {"role": "system", "content": "Extract the invoice fields. Reply with JSON only."}, {"role": "user", "content": text}, , format=SCHEMA, options={"temperature": 0}, data = json.loads reply "message" "content" missing = key for key in SCHEMA "required" if key not in data if missing: raise ValueError f"model output lacks {missing}" return data if name == " main ": path = sys.argv 1 if len sys.argv 1 else "data/invoice.txt" with open path, encoding="utf-8" as handle: print json.dumps extract invoice handle.read , indent=2 Sample output: { "vendor": "Northwind Print Co.", "total": 340.5, "currency": "USD", "due date": "2026-12-05", "line items": {"description": "Business cards x500", "amount": 84.0}, {"description": "Poster printing x12", "amount": 216.5}, {"description": "Rush fee", "amount": 40.0} } Check the math in code before you trust it: line items should sum to the total. A schema guarantees shape, never truth. If you later need to grade many extractions automatically, a local LLM-as-a-judge harness https://shattered.io/llm-as-a-judge-local-eval-harness-12-steps-2026/ gives you a repeatable scoring loop. Step 8: Let the model call your functions Tool calling lets a model ask your program to run a function. You describe the functions in a tools array, the model replies with tool calls instead of text when it wants one, you run the function, and you send the result back as a message with the role tool . The model then writes its final answer. The model never executes anything. Your code stays in charge, which is the property that makes this safe enough to use. Save the following as tools demo.py . It exposes two tools: one adds numbers, and one reads a note from the data/ folder. The note reader resolves the path and refuses anything outside data/ , because a model that can pass any path can read any file your user can. python import json from pathlib import Path from client import Ollama DATA = Path "data" .resolve def add numbers a, b : return a + b def read note name : target = DATA / name .resolve if DATA not in target.parents: return "error: path outside data/" if not target.is file : return "error: no such note" return target.read text encoding="utf-8" :2000 REGISTRY = {"add numbers": add numbers, "read note": read note} TOOLS = { "type": "function", "function": { "name": "add numbers", "description": "Add two numbers and return the sum", "parameters": { "type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}, "required": "a", "b" , }, }, }, { "type": "function", "function": { "name": "read note", "description": "Read a text note from the data folder by file name", "parameters": { "type": "object", "properties": {"name": {"type": "string"}}, "required": "name" , }, }, }, def run question, max rounds=4 : llm = Ollama messages = {"role": "user", "content": question} for in range max rounds : reply = llm.chat messages, tools=TOOLS, options={"temperature": 0} message = reply "message" messages.append message calls = message.get "tool calls" or if not calls: return message "content" for call in calls: name = call "function" "name" args = call "function" "arguments" func = REGISTRY.get name try: result = func args if func else f"error: unknown tool {name}" except TypeError as exc: result = f"error: bad arguments {exc} " messages.append {"role": "tool", "tool name": name, "content": str result } return "stopped: too many tool rounds" if name == " main ": print run "What is 1284.5 plus 15.25? Use the tool." print run "Read invoice.txt and tell me who the vendor is." Run python tools demo.py . A healthy run prints the result of the tool call, then the model’s final answer. The max rounds cap matters: without it, a confused model can loop on the same tool call forever. Not every model supports tools. Ollama returns an error for models that do not declare the capability, and the troubleshooting section covers that failure. Step 9: Create embeddings and search your own notes Embeddings turn text into vectors so that similar meanings land close together. The /api/embed endpoint accepts a string or a list of strings and returns one vector per input, which lets you embed a whole folder in one request. Use a dedicated embedding model such as embeddinggemma . A chat model is the wrong tool for this job. Create two small notes to search. cat data/runbook.txt <<'EOF' To rotate the database password, update the secret in the vault and restart the API pods one at a time. If disk usage on the log server passes 85 percent, delete rotated logs older than 14 days. The staging cluster is rebuilt every Sunday night from the main branch. EOF Save this as search.py . It splits each file into paragraphs, embeds them in one batch, embeds the query, and ranks by cosine similarity using only the standard library. python import math import sys from pathlib import Path from client import Ollama def cosine a, b : dot = sum x y for x, y in zip a, b norm = math.sqrt sum x x for x in a math.sqrt sum y y for y in b return dot / norm if norm else 0.0 def load chunks folder="data" : chunks = for path in sorted Path folder .glob " .txt" : for para in path.read text encoding="utf-8" .split "\n\n" : if para.strip : chunks.append path.name, para.strip return chunks def search query, top=3 : llm = Ollama chunks = load chunks vectors = llm.embed text for , text in chunks query vec = llm.embed query 0 scored = cosine query vec, vec , name, text for vec, name, text in zip vectors, chunks return sorted scored, reverse=True :top if name == " main ": question = " ".join sys.argv 1: or "what do I do when the log disk fills up" for score, name, text in search question : print f"{score:.3f} {name}: {text :80 }" Sample output for the default query: 0.712 runbook.txt: If disk usage on the log server passes 85 percent, delete rotated logs older than 1 0.402 runbook.txt: To rotate the database password, update the secret in the vault and restart the A 0.371 runbook.txt: The staging cluster is rebuilt every Sunday night from the main branch. The right paragraph wins even though the query shares almost no words with it. That is the point of embeddings. Scores vary by model, so compare them within one model and never across two. This script re-embeds everything on each run. A real application should cache vectors on disk and re-embed only files that changed. Steps 10 to 12: Model Management, OpenAI Compatibility and Hardening Step 10: Manage models through the API and a Modelfile Everything the CLI does for model housekeeping has an API twin, so a deployment script never needs to shell out. The reference table below maps each endpoint from Ollama’s API reference to the job it does in this tutorial. | Endpoint | Method | Job | Used in | |---|---|---|---| | /api/version | GET | Server health and version | Step 1 | | /api/generate | POST | Single-prompt completion, model unload | Steps 3, 6 | | /api/chat | POST | Multi-turn chat, structured output, tools | Steps 4 to 8 | | /api/embed | POST | Vectors for one or many strings | Step 9 | | /api/tags | GET | List installed models | Step 10 | | /api/ps | GET | List models loaded in memory | Step 6 | | /api/show | POST | Inspect one model and its Modelfile | Step 10 | | /api/create | POST | Build a custom model from a base | Step 10 | | /api/copy | POST | Duplicate a model under a new name | Step 10 | | /api/pull | POST | Download a model | Step 2 | | /api/delete | DELETE | Remove a model from disk | Step 10 | Try the read-only calls first. curl -s http://localhost:11434/api/tags | jq -r '.models .name' curl -s http://localhost:11434/api/show -d '{"model": "gemma4"}' | jq '{details, capabilities}' The /api/show response includes the model’s details and, on current releases, a capabilities list. Check that list before you build a feature on tools or vision, because a model that lacks the capability will refuse the request. Now create a custom model. The API form takes a from base model plus the settings you would otherwise put in a Modelfile. Set stream to false so you get one status object instead of a progress feed. curl -s http://localhost:11434/api/create -d '{ "model": "terse-gemma", "from": "gemma4", "system": "Answer in at most two sentences. Never use bullet points.", "parameters": {"temperature": 0.2}, "stream": false }' curl -s http://localhost:11434/api/copy -d '{"source": "terse-gemma", "destination": "terse-gemma-backup"}' curl -s -X DELETE http://localhost:11434/api/delete -d '{"model": "terse-gemma-backup"}' The CLI route does the same job with a file. Ollama’s Modelfile reference lists the instructions: FROM required , PARAMETER , TEMPLATE , SYSTEM , LICENSE , MESSAGE , REQUIRES and CAPABILITY . Save this as Modelfile , then build and test it. FROM gemma4 PARAMETER temperature 0.2 PARAMETER num ctx 8192 SYSTEM """You are a release-notes writer. Answer in at most two sentences.""" ollama create release-writer -f Modelfile ollama run release-writer "Summarize: fixed a crash when the cache was empty" The same file format imports weights you downloaded yourself. Ollama’s reference says a FROM line can point at a GGUF file by absolute path or by a path relative to the Modelfile. That matters for new releases that have not reached the library yet. NVIDIA’s documentation, for instance, lists a Q4 K M GGUF checkpoint for Nemotron 3.5 Bolt E4B 4GB. After you download a GGUF file from the publisher, a Modelfile with FROM ./your-model.gguf turns it into a named model you can call through every endpoint in this tutorial. The llama.cpp and Open WebUI tutorial https://shattered.io/llama-cpp-server-open-webui-gguf-12-steps-2026/ explains how GGUF files and their quantization levels differ, which helps you choose the right file. Step 11: Call Ollama from the OpenAI client The compatibility layer lets existing code talk to a local model by changing the base URL. Ollama’s docs list the base as http://localhost:11434/v1 . The OpenAI client insists on an API key string, so pass any placeholder. Ollama ignores it for local calls. curl -s http://localhost:11434/v1/models | jq -r '.data .id' Save this as compat check.py . python import os from openai import OpenAI client = OpenAI base url="http://localhost:11434/v1", api key="ollama" model = os.environ.get "OLLAMA MODEL", "gemma4" resp = client.chat.completions.create model=model, messages= {"role": "user", "content": "Reply with the single word: ready"} , temperature=0, print resp.choices 0 .message.content print m.id for m in client.models.list .data Compatibility covers the common paths, but it is not a full clone. Ollama’s documentation lists supported features on its OpenAI compatibility page, and anything not listed there should be tested before you rely on it. When a feature behaves oddly through /v1 , call the native /api/chat endpoint instead. The native API always gets new features first, as Step 7’s schema support and Step 8’s tool messages show. Step 12: Lock down exposure and add a health check By default the server listens on loopback only, so other machines cannot reach it. Ollama’s FAQ explains how to change that with the OLLAMA HOST variable, for example OLLAMA HOST=0.0.0.0:11434 . On Linux with systemd, you run systemctl edit ollama.service , add the variable under the Service heading as an Environment line, save, and restart the service. On macOS, the FAQ uses launchctl setenv OLLAMA HOST "0.0.0.0:11434" and a restart of the app. Here is the catch. Ollama’s documentation states that local requests do not need an API key. So the moment you bind to 0.0.0.0 , anyone who can route to the port can run your models, pull new ones and, if the machine allows it, delete or create models. Treat that port like a database port. Service Environment="OLLAMA HOST=127.0.0.1:11434" Keep the loopback binding and put a reverse proxy in front when other machines need access. Add TLS and token checks there. A firewall rule that allows only the proxy host is the second layer. If browser code on another origin needs to call the API, set OLLAMA ORIGINS to the exact origins rather than a wildcard. Finally, write a health check you can run after every upgrade. Save this as smoke test.py . It exercises each capability from the earlier steps, prints a pass or fail line with timing, and exits non-zero on any failure, which makes it usable in cron or CI. python import sys import time from client import Ollama, tokens per second from extract import extract invoice from search import search from tools demo import run as run tools def installed llm : names = llm.models for name in names: if name.split ":" 0 == llm.model.split ":" 0 : return name raise RuntimeError f"{llm.model} not in {names}" def check label, fn : start = time.perf counter try: detail, status = fn , "PASS" except Exception as exc: detail, status = str exc :70 , "FAIL" print f"{status:<5} {label:<18} {time.perf counter - start:6.1f}s {detail}" return status == "PASS" def main : llm = Ollama results = check "server version", llm.version , check "model installed", lambda: installed llm , check "generate", lambda: f"{tokens per second llm.generate 'Say hi', options={'num predict': 32} :.1f} tok/s" , check "chat", lambda: llm.chat {"role": "user", "content": "Say ok"} "message" "content" :40 , check "structured output", lambda: extract invoice open "data/invoice.txt", encoding="utf-8" .read "vendor" , check "tool calling", lambda: run tools "What is 2 plus 40? Use the tool." :40 , check "embeddings", lambda: f"{len llm.embed 'hello' 0 } dims" , check "semantic search", lambda: search "log disk full" 0 1 , print f"{sum results }/{len results } checks passed" sys.exit 0 if all results else 1 if name == " main ": main Read the Numbers: Latency, Throughput and Memory Every non-streaming reply, and the final chunk of a stream, carries timing fields. They turn “it feels slow” into something you can compare across models, quantization levels and context sizes. All durations are in nanoseconds. | Field | Unit | What it tells you | |---|---|---| | total duration | ns | The whole request, including model load | | load duration | ns | Time spent loading the model into memory | | prompt eval count | tokens | How many prompt tokens the model processed | | prompt eval duration | ns | Time spent reading the prompt | | eval count | tokens | How many tokens the model generated | | eval duration | ns | Time spent generating those tokens | Two speeds matter, and they behave differently. Prompt processing speed is prompt eval count divided by prompt eval duration . It governs how long a big document takes before the first token appears. Generation speed is eval count divided by eval duration . It governs how fast text streams once it starts. A retrieval app that pastes long notes into every prompt cares most about the first number, while a chat app cares about the second. Measure fairly. Warm the model first so load time stays out of the result. Use a fixed prompt, set temperature to 0 and cap num predict so each run does the same work. Then report the median of several runs, because the first request after any change is an outlier. Save this as bench.py . python import statistics import sys from client import Ollama, tokens per second PROMPT = "Write a 120-word explanation of how TLS handshakes work." def bench runs=5 : llm = Ollama llm.generate "warm up", options={"num predict": 8} speeds, loads = , for in range runs : reply = llm.generate PROMPT, options={"temperature": 0, "num predict": 200} speeds.append tokens per second reply loads.append reply "load duration" / 1e9 return speeds, loads if name == " main ": runs = int sys.argv 1 if len sys.argv 1 else 5 speeds, loads = bench runs print f"runs={runs} median={statistics.median speeds :.1f} tok/s min={min speeds :.1f} max={max speeds :.1f}" print f"load time per call: {statistics.median loads :.2f}s" Sample output the figures below are placeholders, since speed depends entirely on your hardware, model and quantization : runs=5 median=NN.N tok/s min=NN.N max=NN.N load time per call: 0.0Xs A load time near zero means the model stayed resident between calls. Several seconds on every call means keep alive is too short, or another model keeps evicting yours. For memory, run ollama ps during the benchmark and compare size with size vram from /api/ps . If size vram is smaller, layers run on the CPU and generation slows. Change one variable per run, whether the model tag, num ctx or the quantization, and keep a short log of results. The Complete Working Project You now hold every file. The finished folder looks like this. ollama-api-lab/ client.py, chat demo.py, stream demo.py extract.py, tools demo.py, search.py compat check.py, bench.py, smoke test.py app.py data/invoice.txt, data/runbook.txt The last file ties the pieces together as a terminal assistant. It streams answers, trims history so the context window never overflows, answers from your notes with /notes , and extracts invoices with /extract . Save it as app.py . python import json from client import Ollama, OllamaError from extract import extract invoice from search import search HELP = "commands: /notes QUESTION, /extract FILE, /quit" def trim history, keep=12 : return history :1 + history 1: -keep: def main : llm = Ollama print f"ollama {llm.version } | model {llm.model}\n{HELP}" history = {"role": "system", "content": "You are a concise assistant."} while True: try: line = input "you " .strip except EOFError, KeyboardInterrupt : break if not line: continue if line == "/quit": break if line.startswith "/extract " : try: with open line 9: .strip , encoding="utf-8" as handle: print json.dumps extract invoice handle.read , llm , indent=2 except OSError, ValueError, OllamaError as exc: print f" error {exc}" continue if line.startswith "/notes " : question = line 7: context = "\n".join text for , , text in search question line = f"Answer using only these notes:\n{context}\n\nQuestion: {question}" history.append {"role": "user", "content": line} history = trim history answer = "" try: for chunk in llm.stream chat history : piece = chunk "message" "content" answer += piece print piece, end="", flush=True except OllamaError as exc: print f"\n error {exc}" history.pop continue print history.append {"role": "assistant", "content": answer} if name == " main ": main Run the health check first, then the assistant. python smoke test.py python app.py Sample smoke-test output timings, speed and vector size depend on your hardware and models : PASS server version 0.0s 0.40.9 PASS model installed 0.0s gemma4:latest PASS generate 2.6s 41.3 tok/s PASS chat 1.1s ok PASS structured output 3.4s Northwind Print Co. PASS tool calling 2.9s The sum of 2 and 40 is 42. PASS embeddings 0.4s NNN dims PASS semantic search 0.9s runbook.txt 8/8 checks passed The dimension count depends on the embedding model, so do not hard-code it anywhere. If every line passes, you have a working local API stack: chat, streaming, structured output, tools and retrieval, all on hardware you control. If one line fails, jump to the troubleshooting list below, which follows the same order. Common Pitfalls Seven mistakes account for most of the support questions that local-model developers raise. 1. Expecting memory from a stateless API. The server keeps no conversation. Resend the history on every chat call and trim it before it passes the context window. 2. Ignoring the 4096-token default. Ollama’s FAQ gives 4096 tokens as the default context window. Long documents and long histories get cut without a loud error, and answers quietly lose their beginnings. Raise num ctx deliberately and watch memory. 3. Parsing a stream as one JSON body. Streaming responses are newline-delimited objects. Read line by line, and treat an error object mid-stream as a failure even though the HTTP status was 200. 4. Binding to 0.0.0.0 without a proxy. Local calls need no key, so an open port is an open door. Keep loopback binding and add authentication at a proxy. 5. Hard-coding a model tag that vanishes. Cloud tags retire. Ollama’s model lifecycle page records that qwen3.5:397b , glm-5.1 and deepseek-v4-flash:0731 were retired, with glm-5.3 , glm-5.2 and deepseek-v4.1-flash listed as replacements. Read the model name from an environment variable and run the smoke test after changes. 6. Testing creation against the hosted API. Creating and deleting models works only on a local server. The hosted API is for inference. 7. Using localhost blindly. Some clients resolve localhost to the IPv6 address first, while the server listens on IPv4 only. Switch to 127.0.0.1 when you see sporadic connection refusals. Troubleshooting Each item below follows the order of the steps, from setup problems to the subtle ones. 1. “Connection refused” on port 11434. The server is not running. Start the desktop app, run systemctl status ollama on Linux, or start ollama serve in a terminal. Then retry curl http://127.0.0.1:11434/api/version . 2. HTTP 404 with “model not found”. The tag is not installed. Run ollama ls , then ollama pull the exact name. Remember that gemma4 and gemma4:latest are the same, while gemma4:31b is a different download. 3. The first request takes a minute or times out. A cold start loads weights from disk. Raise the client timeout the wrapper uses 300 seconds , warm the model with a tiny prompt at startup, and set keep alive high enough to avoid reloads. 4. ollama ps shows 100% CPU. The GPU is not in use. Check Ollama’s hardware support page, update GPU drivers, and confirm the model fits in video memory. A split such as 48%/52% CPU/GPU means it does not fit, so pick a smaller tag or a smaller num ctx . 5. The stream stops with an error object. The model crashed or ran out of memory after the response started. Read the server log journalctl -u ollama -f on Linux , shrink the context, and unload other models with keep alive 0. 6. “does not support tools” HTTP 400 . The model lacks the tools capability. Query /api/show and read capabilities , then switch to a tool-capable model or drop the tools field. Agent frameworks such as the OpenHands local setup https://shattered.io/openhands-local-setup-agent-canvas-ollama-12-steps-2026/ depend on tool calling, so they hit this error first when you pick the wrong model. 7. Structured output comes back truncated or invalid. num predict may be cutting the reply. Raise it, set temperature to 0, simplify the schema, and keep required lists short. 8. Browser code gets a CORS error. Set OLLAMA ORIGINS to the exact page origin, restart the server, and test again. Better still, call the API from your backend. 9. A Docker container cannot reach the host server. localhost inside a container points at the container. Use the host gateway name your platform provides, and bind the server to an interface the container network can reach, with a firewall rule limiting access. 10. /api/create or /api/delete fails. You are probably calling https://ollama.com . Switch the base URL to the local server. 11. The OpenAI client raises an authentication error. Pass any non-empty api key string and confirm the base URL ends in /v1 . 12. Embedding vectors have a different length after a model change. Each embedding model has its own dimension count. Re-embed your whole corpus when you switch models and never mix vectors from two models in one index. Advanced Tips Keep the stable part of every prompt at the front. On November 16, 2026, Ollama described prefix caching for long-horizon agent workloads, meant to reuse earlier conversation and tool-call work. Whatever the exact mechanics in your version, a prompt that starts with the same system message and tool definitions on every call gives any prefix-based cache the best chance to help. Put the changing question last. Tune concurrency deliberately. Ollama exposes server variables such as OLLAMA NUM PARALLEL and OLLAMA MAX LOADED MODELS for parallel requests and resident models. Raise them one at a time and rerun the smoke test, because each extra parallel slot enlarges the memory footprint. When a service needs both a chat model and an embedding model, keep both resident with a long keep alive so neither evicts the other. Pin what you depend on. The Linux installer accepts an OLLAMA VERSION variable, per Ollama’s documentation, so a deployment script can install a known build and upgrade on purpose. A Modelfile REQUIRES line states the minimum Ollama version a custom model needs. Record the model digest from /api/tags in your deploy notes so you can tell when a tag changes underneath you. Send images by adding base64 strings to a message’s images list when a model supports vision, and check capabilities first. Ollama’s CLI reference also shows speech: ollama run phonon "Hello from Ollama." generates speech, and redirecting output with speech.wav writes a WAV file. Speech needs v0.40.9 or later. Point your editor tooling at the same server. ollama launch starts supported integrations that include OpenCode, Claude Code, Codex, VS Code and Droid, per the CLI reference. For agent-style coding on local models, the Goose local setup https://shattered.io/goose-ai-coding-assistant-local-ollama-12-steps-2026/ and the Aider and Git workflow https://shattered.io/aider-local-ai-coding-assistant-ollama-git-2026/ both reuse the endpoint you just verified. FAQ What port does the Ollama API use? Port 11434 on localhost, for the native /api routes and the OpenAI-compatible /v1 routes alike. Change it with the OLLAMA HOST variable, for example OLLAMA HOST=127.0.0.1:8080 , and restart the server. Does the Ollama API need an API key? Not for local use. Ollama’s API introduction says cloud requests need an API key while local requests do not. The hosted API at ollama.com does require a Bearer token. OpenAI client libraries still want a placeholder string for local calls. Should I use /api/generate or /api/chat? Use /api/chat for almost everything. It handles message roles, structured output, tools and multi-turn history. Keep /api/generate for one-shot prompts, simple tests and the unload trick with keep alive set to 0. Can I use the OpenAI Python library with Ollama? Yes. Set base url to http://localhost:11434/v1 , pass any non-empty API key, and use a model name from ollama ls . Test any feature beyond basic chat before depending on it, and fall back to the native API when something differs. Is my data sent to Ollama when I call the local API? Not for local models. Ollama’s local FAQ says it does not see your prompts or data when you run locally. Models tagged for the cloud run on hosted infrastructure, so check the tag before you send anything sensitive. How do I stop the model from forgetting earlier messages? Send the whole message history on every chat request and append each assistant reply before the next question. Trim old turns when the history nears the context window, or raise num ctx within your memory budget. Will code written against the API break after an upgrade? Unlikely. Ollama says the API is not strictly versioned but is expected to remain stable and backward compatible, with rare deprecations announced in release notes. Run smoke test.py after every upgrade to catch the exceptions. How do I use a newer open model that is not in the Ollama library yet? Download its GGUF checkpoint from the publisher, point a Modelfile FROM line at the file, run ollama create , and call the new name through any endpoint. Confirm the model’s chat template and capabilities with /api/show before relying on tools or vision.