Running Ollama locally has become the simplest way into self-hosted LLMs: one command to install it, another to download a model, and an HTTP API ready to respond. It is one of the most followed open source projects on GitHub, used everywhere from a developer’s laptop to a team GPU server. As soon as an Ollama local LLM serves several people, it deserves to be monitored like any other service: metrics, logs and traces.
This simplicity has a downside. As long as Ollama remains a personal tool, nobody wonders how long a model takes to load or why an answer takes twelve seconds to arrive. As soon as it becomes a service that other applications depend on, it has to be treated like any other production component: you need to know whether it responds, how fast, with what GPU memory consumption, and for which callers.
This article introduces the project, installs it step by step, then shows how to monitor it: what the API exposes natively, how to collect the logs, how to produce OpenTelemetry traces on the client side and how all of this flows into an observability tool such as Dynatrace.
Ollama is a language model runtime that wraps model download, quantisation and serving behind a command-line interface and a REST API. Internally it relies on llama.cpp and the GGUF format, which lets it run on CPU, on NVIDIA or AMD GPUs, and on Apple Silicon chips.
Its strengths for a technical team:
/api/generate, /api/chat, /api/tags…) and an OpenAI-compatible API under /v1, so existing SDKs can be reused;
Its limits are just as clear: Ollama aims for simplicity more than maximum throughput. To serve many concurrent users on datacentre GPUs, a server such as vLLM in production is often a better fit.
On Linux, the official script installs the binary and creates a systemd service:
curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama
On macOS and Windows, a graphical installer is available. In a container, the official image is used like this (add --gpus=all if the NVIDIA Container Toolkit is installed):
docker run -d --name ollama \
-v ollama:/root/.ollama \
-p 11434:11434 \
ollama/ollama
ollama pull llama3.2
ollama run llama3.2 "Explain what an SLO is in two sentences."
ollama ps # models loaded in memory, CPU or GPU
ollama list # models present on disk
Available names and tags change constantly: check the model library on the project’s website before hard-coding a name in a script.
By default, the server listens on port 11434 of the local machine. A non-streamed call returns, in addition to the text, timing information that is very useful for monitoring:
curl -s http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Why is the sky blue?",
"stream": false
}' | jq '{total_duration, load_duration, prompt_eval_count, eval_count, eval_duration}'
Durations are expressed in nanoseconds. load_duration reveals the cost of a cold load; eval_count divided by eval_duration gives the generation throughput in tokens per second.
To expose Ollama to other machines, the OLLAMA_HOST variable sets the listening address. Other variables control how long a model stays in memory or the degree of parallelism. Their list and exact behaviour change from one version to the next: refer to the FAQ in the official Ollama documentation. The API has no built-in authentication: put a reverse proxy with access control in front of it, and never expose it directly on the internet.
Monitoring a local LLM relies on four families of signals. Ollama provides only some of them natively; the rest is built around it.
A simple call to /api/tags or to the server root is enough as an availability probe. The /api/ps endpoint lists the loaded models, their size and the share placed in GPU memory: it is the API equivalent of ollama ps, easy to query from a synthetic monitor.
To our knowledge, Ollama does not expose a native Prometheus endpoint at the time of writing; check the release notes, as the topic is regularly discussed. Useful metrics are therefore calculated from API responses (total latency, load time, input and output tokens, throughput) and from infrastructure metrics. On the GPU side, NVIDIA’s DCGM exporter or a host agent provides card utilisation, memory and temperature.
Installed as a service, Ollama writes to the systemd journal; in a container, to standard output:
journalctl -u ollama -f
docker logs -f ollama
These logs show, among other things, GPU detection, model and memory errors. Verbose mode is enabled through a debug variable described in the documentation. A collection agent (OneAgent, or an OpenTelemetry Collector with the filelog or journald receiver) centralises them.
The richest data lives with the caller. The OpenLLMetry project publishes OpenTelemetry instrumentations for several providers, including Ollama, which produce one span per call with the model, the tokens consumed and the duration, following the GenAI semantic conventions. Example in Python:
pip install ollama opentelemetry-sdk opentelemetry-exporter-otlp \
opentelemetry-instrumentation-ollama
python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.ollama import OllamaInstrumentor
import ollama
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
OllamaInstrumentor().instrument()
reponse = ollama.chat(model="llama3.2",
messages=[{"role": "user", "content": "Hello"}])
print(reponse["message"]["content"])
Check the exact package and class names in the OpenLLMetry repository: these instrumentations change quickly. We compare this approach with Langfuse in our article on Langfuse and OpenLLMetry.
The OTLP exporter is configured through the standard OpenTelemetry variables. For direct sending to Dynatrace SaaS, the environment’s OTLP endpoint and an API token with the appropriate ingest permissions (traces, metrics, logs) are enough:
export OTEL_SERVICE_NAME="assistant-interne"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<environment>.live.dynatrace.com/api/v2/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Api-Token%20<token>"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
In a more serious deployment, it is better to go through an OpenTelemetry Collector, which receives traces and logs, adds host and Kubernetes attributes, then exports to the backend. The spans then appear in the distributed traces view, attached to the calling service, and the host running Ollama is correlated with its GPU metrics. The principle is the same as the one described in end-to-end OpenTelemetry: the value comes from the link between the model call, the service and the infrastructure.
Few dashboards are really useful: p50 and p95 latency per model, tokens generated per minute, share of calls with a cold load, GPU memory used, error rate. One alert on p95 latency and another on memory saturation cover the essentials.
For internal use, prototypes or sensitive data, yes, provided you accept smaller models and size the hardware appropriately. For heavy concurrent traffic, a throughput-oriented server is preferable.
Not natively, to our knowledge. You combine the durations returned by the API, the application’s OpenTelemetry traces and the host’s GPU metrics. Check the latest releases, as this may change.
The ollama ps command or the /api/ps endpoint shows the CPU/GPU split for each loaded model; the startup logs also show the GPUs detected.
Ollama makes local inference almost trivial, but a shared LLM is still a production service. By combining API probes, centralised logs, GPU metrics and application-side OpenTelemetry traces, you get a complete picture, ready to be used in Dynatrace or another backend. To see where Ollama fits among the other building blocks, read our comparison of self-hosted open source AI stacks, and to add a chat interface to it, our Open WebUI guide.
span.observer is an independent community blog, not affiliated with Dynatrace LLC or the projects mentioned. Trademarks belong to their respective owners. Tools change quickly: check commands and options in their official documentation.
Originally published on span.observer. French version: https://span.observer/fr/ollama-llm-local-supervision/.