![llama.cpp and Open WebUI: 12 Steps, 60 Min [2026]](https://shattered.io/wp-content/uploads/2026/10/llama-cpp-server-open-webui-gguf-12-steps-2026-gen.webp)
Xiaomi retired the MiMo-V2.5-Pro and MiMo-V2.5 API model names on October 21, 2026, and the old IDs stopped working at 10:00 Beijing time, TechNode reported. Every app still calling them started returning errors. A GGUF file on your own disk can’t be switched off that way, and that gap is the plainest reason to learn how to run an LLM locally.
OpenAI ran a milder version of the same play on October 14, when its model release notes show GPT-5.5 leaving ChatGPT while API access stayed untouched. Hosted models change on the vendor’s calendar. Open weights served from your own machine change when you decide they should.
llama.cpp is the engine under most local setups. It loads GGUF model files and runs them on CPUs, NVIDIA and AMD GPUs, and Apple silicon. Its llama-server binary speaks the OpenAI API, so editors, scripts and chat front ends connect without custom code. Add Open WebUI and you get a familiar browser chat on hardware you own. An hour of work takes a bare machine to a private chat server, including a GGUF file you quantize yourself. Readers who prefer a friendlier wrapper can follow our Ollama setup on an RTX 5090 and return here when they need finer control.
Don't miss new tech stories on Google
Add Shattered once in the Google app and our stories show up in your news suggestions.
Why Run an LLM Locally With llama.cpp Instead of Calling an API #
The first benefit is stability. A GGUF file you downloaded last month answers the same way today, with the same chat template and the same cost, which is electricity. No vendor can rename the model, retire its ID or quietly retune it. For regression tests, regulated workflows and anything that has to be reproducible, that matters more than a leaderboard rank.
The second is data control. Prompts, documents and chat history stay on your disk. Treat the privacy claim with some care, though. OpenTools made the point in an October 19 piece on a Windows app that puts llama.cpp in the system tray: local privacy still depends on the model and the tools around it. A local model wired to a web-search plugin or a remote tool server sends data out just as surely as an API call would. Audit the tools before you trust the label.
Plenty of developers already work this way. In the 2025 Stack Overflow Developer Survey, 17.8% of respondents said they had used Meta Llama models for development work in the past year. The same survey’s AI section found that 51.1% of developers building AI agents used Ollama, a wrapper that sits on top of the same kind of engine this tutorial uses directly.
Cost is the third reason. A GPU you already own turns tokens into a fixed expense. Agents that loop over a codebase all day, or batch jobs that summarize thousands of documents, chew through per-token budgets fast. Locally they cost power and wear.
Local is not free, and it is not magic. You pay in hardware, setup time and model size. Hosted frontier models still lead on the hardest reasoning and coding tasks, and no 4-bit quantization of a mid-sized model changes that. The realistic pattern is hybrid: local models for private, high-volume or latency-sensitive work, hosted models for the tasks where raw capability decides the outcome. Building the local half well is what the next twelve steps do.
llama.cpp vs Ollama vs vLLM: Picking a Local Runtime #
Three runtimes cover almost every self-hosted LLM setup, and they solve different problems. llama.cpp is a C and C++ inference engine built around the GGUF format. It runs on a laptop CPU, a Mac, a gaming GPU or a server, and it exposes nearly every knob: context size, layer offload, KV cache precision, speculative decoding, expert offload for mixture-of-experts models.
Ollama wraps a pinned llama.cpp build in a daemon with a model registry and one-line pulls. It trades control for convenience. vLLM targets GPU servers. It serves safetensors checkpoints with continuous batching and wins when many users hit one model at the same moment.
| Factor | llama.cpp | Ollama | vLLM |
|---|---|---|---|
| Install effort | One binary via brew or winget, a Docker image, or a source build | Installer plus a background daemon | Python environment or container with a CUDA stack |
| Native model format | GGUF | GGUF through its own registry and Modelfiles | Safetensors, plus quantized checkpoints such as AWQ, GPTQ and FP8 |
| Hardware reach | CPU, CUDA, ROCm, Vulkan, Metal and more | Same engine underneath, fewer exposed options | GPU-first |
| Many simultaneous users | Parallel slots, modest scale | Limited | Continuous batching, strongest |
| Split a big model across GPU and RAM | Yes, per layer and per expert | Automatic, little control | Limited |
| Flag-level control | Full | Partial, via Modelfile | Extensive engine arguments |
| Best fit | Mixed hardware, tinkering, edge boxes | Fast local chat | Production serving on GPUs |
Pick llama.cpp when you want to run on mixed hardware, squeeze a large model into limited memory by pushing layers into system RAM, or control exactly which build and flags run. Pick Ollama when you want a model answering in two minutes and don’t care about flags. Pick vLLM when concurrency, not memory, is the bottleneck. Our vLLM and Ollama walkthrough on a DGX Spark covers that server-class end of the range.
This tutorial stays with llama.cpp because its server mode is the part the other tools hide. Learn it once and Ollama’s behavior stops being mysterious. One practical warning: the project ships new builds almost daily. Third-party release trackers listed builds b11870 and b11871 on October 25, 2026, so record the exact build number next to every benchmark you take.
Prerequisites: Hardware, Software and Versions #
Memory decides everything. The file you load has to fit in VRAM, unified memory or system RAM, with room left for the KV cache that grows with every token of context. OpenAI says gpt-oss-20b runs within roughly 16 GB of memory and gpt-oss-120b within roughly 80 GB, thanks to native MXFP4 quantization. Those two figures anchor the rest of this guide. A 16 GB GPU, a 16 GB Apple silicon Mac or a desktop with 32 GB of RAM can handle the smaller model. The larger one wants a workstation, a unified-memory machine or a great deal of system RAM.
You don’t need a flagship card. NVIDIA’s push toward 24 GB and larger RTX cards for local AI shows where consumer hardware is heading, yet a CPU-only machine runs every step below with a smaller model. It just answers slowly.
| Component | Minimum | Notes |
|---|---|---|
| Operating system | Not listed | Linux exposes the most backends and flags |
| Memory for the main model | 16 GB VRAM or unified memory for gpt-oss-20b | 8 GB is enough for small 4B to 8B models at 4-bit |
| NVIDIA driver | A current release that matches your CUDA toolkit | Check with nvidia-smi |
| CUDA Toolkit | 12.x or 13.x | Needed only for source builds, prebuilt binaries and Docker images bundle the runtime |
| CMake | Not listed | Source builds only |
| C++ compiler | Any C++17 compiler (GCC, Clang or MSVC 2022) | Install build-essential on Ubuntu |
| Python | Not listed | Hugging Face CLI, the convert script, and the pip route for Open WebUI |
| Docker | Engine with the Compose v2 plugin | Add the NVIDIA Container Toolkit for GPU containers |
| llama.cpp | Build b11581 or newer | b11581 from October 13, 2026 is the newest development build confirmed in the sources checked |
| Disk space | Model file plus Docker images | Keep a second copy of the model if you plan to re-quantize |
Open WebUI moves quickly too. Third-party trackers list v0.12.4 as the newest release on October 25, 2026, after v0.12.3 on October 16, though I could not confirm that from an official page. Pin the tag you test rather than trusting a floating one. Any llama.cpp build from mid-October on should work with every command here.
Which Open-Weight Models Run in llama.cpp Today #
Not every new model is a local model. Vendors announce hosted APIs, previews and genuine downloads in the same breath, and the headlines blur them. Three questions sort the real options: are the weights downloadable, does llama.cpp support the architecture, and does a GGUF conversion exist or can you make one?
gpt-oss passes all three, which is why it anchors this tutorial. OpenAI released gpt-oss-120b and gpt-oss-20b under the Apache 2.0 license, and the weights are free to download from Hugging Face with MXFP4 quantization built in. The models handle text only and support function calling and structured outputs. They aren’t served through the OpenAI API or ChatGPT, so you run them yourself or through a hosting provider. As of this writing no newer gpt-oss release is confirmed. It is still the easiest high-quality entry point because the memory target is so small.
Alibaba’s Qwen team publishes open-weight models on Hugging Face, ModelScope and Kaggle, and a September overview reportedly placed the Qwen3.8 generation in August 2026, with a Qwen3.8 27B model appearing in an October 16 comparison. Treat that as reported, not confirmed. Read the model card for a GGUF build before you plan around it.
GLM and MiMo need more caution. Reports describe GLM-5.3-Flash as a 320-billion-parameter mixture-of-experts model with 18 billion active parameters, but sources conflict on whether its weights are open, and Z.ai’s newest release, GLM-Legal, is a gated beta API. Xiaomi says it open-sourced the MiMo-V2.6 family on September 22, yet none of the sources I checked confirm an official GGUF or llama.cpp support. DeepSeek V4.1-Flash is reported with MIT-licensed weights, but that also needs confirmation from the official repository. For now, plan your local stack around models whose GGUF files you can actually open.
Release notes for llama.cpp v0.7.0 reportedly added support for several newer architectures in mid-October, so the gap between a model launch and local support keeps shrinking. For a broader view of which models are worth down, see our October ranking of open-source LLMs. Before you commit disk space to any of them, run this checklist.
| Check | How to do it | Why it matters |
|---|---|---|
| License | Read the model card and the LICENSE file | Apache 2.0 and MIT allow commercial use, custom licenses may not |
| Weight format | Look for .gguf files or safetensors shards | Safetensors need a conversion step first |
| Architecture support | Search the llama.cpp release notes and issues for the model name | An old build fails with an unknown architecture error |
| Memory at your quant | File size plus KV cache at your context length | A model that loads but swaps to disk is unusable |
| Chat template | Check the tokenizer metadata in the GGUF | A missing template produces garbled turns |
GGUF and Quantization: What Q4_K_M Actually Means #
GGUF is a single-file container. It holds the tensors, the tokenizer, the chat template and the model’s metadata, so one file is all llama.cpp needs. Quantization shrinks the tensors by storing each weight in fewer bits. The names look cryptic, but the pattern is simple: the number is the approximate bit width, K means a block-wise scheme that mixes precisions across layers, and the trailing S, M or L picks a small, medium or large variant of that mix.
| Format | Bit class | Quality | Typical use |
|---|---|---|---|
| F16 or BF16 | 16-bit | Reference | Source for conversion, quality baseline |
| Q8_0 | 8-bit | Nearly lossless | When memory is plentiful |
| Q6_K | 6-bit class | Very close to Q8_0 | Quality-first on mid-size GPUs |
| Q5_K_M | 5-bit class | Close to Q6_K | Good balance for coding models |
| Q4_K_M | 4-bit class | Small, visible loss on hard tasks | The default pick for most people |
| Q3_K_M and IQ3 variants | 3-bit class | Noticeable degradation | Only when nothing else fits |
| IQ4_XS and other IQ types | About 4-bit | Better than plain Q4 at similar size, needs an importance matrix | Squeezing the last gigabyte |
| MXFP4 | About 4-bit, microscaling floating point | Native format for gpt-oss weights | Run as shipped, don’t re-quantize |
The memory math is plain arithmetic: billions of parameters times bits per weight, divided by eight, gives gigabytes for the weights alone. K-quants mix bit widths, so real files run a little larger than the 4-bit figures below.
| Parameters | Weights at 8-bit | Weights at roughly 4-bit |
|---|---|---|
| 8 billion | About 8 GB | About 4 to 5 GB |
| 14 billion | About 14 GB | Not listed |
| 30 billion | About 30 GB | About 15 to 18 GB |
| 70 billion | About 70 GB | About 35 to 42 GB |
| 120 billion | About 120 GB | About 60 to 72 GB |
Those are weights only. Add the KV cache, which scales with context length and the number of parallel slots, plus a gigabyte or two of overhead. The 120-billion row lines up with OpenAI’s claim that gpt-oss-120b fits in about 80 GB once MXFP4 is applied.
An importance matrix, or imatrix, improves low-bit quants. You run the full-precision model over a calibration text, record which weights influence the output most, and let the quantizer protect those. Step 10 does this by hand. Models that already ship in a compact native format, like gpt-oss in MXFP4, gain nothing from a second round of quantization. Requantizing an already quantized file only compounds the error.
Steps 1 to 3: Budget Memory, Install llama.cpp, Download a GGUF #
Step 1: Budget your memory
Find out what you have before you download anything. Note the VRAM on a discrete GPU, the unified memory on a Mac, and the system RAM everywhere else. The budget rule is simple: model file size plus KV cache plus about 10 percent headroom must stay under the memory pool you offload to.
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv
free -h
sysctl -n hw.memsize
Get-CimInstance Win32_ComputerSystem | Select-Object TotalPhysicalMemory
Step 2: Install llama.cpp
Package managers are the fastest route on Mac and Windows. On Linux with an NVIDIA card, a source build gives you the right CUDA kernels for your exact GPU. Swap the CMake flag for other hardware: GGML_VULKAN for Vulkan, GGML_HIP for AMD ROCm. Metal is on by default on Apple silicon.
brew install llama.cpp
winget install llama.cpp
sudo apt install -y build-essential cmake git curl libcurl4-openssl-dev
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
export PATH="$PWD/build/bin:$PATH"
llama-server --version
The version command prints a build number and a commit hash. Write both down. Output looks like this, with your own numbers in place of the placeholders:
version: NNNNN (abc1234)
built with GNU X.Y.Z for Linux x86_64
If you’d rather not compile, the project publishes Docker images. The CUDA server image is ghcr.io/ggml-org/llama.cpp:server-cuda, and the Compose stack in Step 12 uses it.
Step 3: Download a GGUF model
The Hugging Face command-line tool pulls exactly the files you ask for. The ggml-org organization on Hugging Face publishes GGUF builds of popular releases, and gpt-oss-20b is among them at the time of writing. If the repository name has changed, open the model page and check its quantizations list instead. Gated models need a token, which you create in your Hugging Face account settings and register with hf auth login.
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -U "huggingface_hub[cli]"
hf download ggml-org/gpt-oss-20b-GGUF --include "*.gguf" --local-dir models/gpt-oss-20b
ls -lh models/gpt-oss-20b
Expect a single .gguf file in the folder, and compare its size against your memory budget from Step 1. llama-server can also fetch models itself with its -hf flag, and the Compose stack relies on that. For local experiments the explicit download is easier to inspect and back up.
Steps 4 to 7: Smoke Test, Launch the Server, Call the API, Tune Flags #
Step 4: Run a one-shot smoke test
Before you start a server, confirm the model loads and generates. llama-cli enters an interactive conversation mode by default, so pass -no-cnv for a single completion. The -ngl 99 flag asks for every layer on the GPU, and llama.cpp quietly caps it at the real layer count.
MODEL=$(ls models/gpt-oss-20b/*.gguf | head -n 1)
llama-cli -m "$MODEL" -ngl 99 -c 4096 -no-cnv -n 128 -p "Explain GGUF in two sentences."
Scan the startup log for the offload line. It tells you whether the GPU is doing the work:
load_tensors: offloaded N/N layers to GPU
...
GGUF is a single-file format that stores a model's weights together with ...
llama_perf_context_print: eval time = ... (... ms per token, XX.XX tokens per second)
If the offloaded count reads 0/N, you built or installed a CPU-only binary. Fix that now, because every later benchmark depends on it.
Step 5: Launch llama-server
Now the real thing. Pass the host and port explicitly. Release notes for a recent llama.cpp tag reportedly changed the default port to 9931, and whether or not that is accurate for your build, an explicit –port means a build upgrade can’t move your endpoint.
llama-server -m "$MODEL" \
--alias gpt-oss-20b \
--host 127.0.0.1 --port 8080 \
-c 16384 -ngl 99 \
--jinja --parallel 2 --metrics
The server also serves a small built-in chat page at its root address, handy for a quick check.
main: server is listening on http://127.0.0.1:8080
main: starting the main loop...
Step 6: Call the OpenAI-compatible API
Open a second terminal. The health route answers without authentication, the models route lists your alias, and the chat route behaves like the hosted API. Give reasoning models a generous max_tokens, because the hidden reasoning tokens count against it.
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/models
curl -s http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-oss-20b","messages":[{"role":"user","content":"Name three uses of a local LLM."}],"max_tokens":512}'
{"status":"ok"}
{"object":"list","data":[{"id":"gpt-oss-20b","object":"model", ...}]}
{"choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"1. Private document search ...","reasoning_content":"..."}}],
"usage":{"prompt_tokens":N,"completion_tokens":N,"total_tokens":N},
"timings":{"prompt_per_second":XX.X,"predicted_per_second":XX.X}}
The timings block is gold for benchmarking, and Step 12 reads it. Python clients use the official OpenAI SDK pointed at your address. The API key can be any string until you enable authentication. Tools built on the same protocol, such as the Aider local coding assistant, connect through the same base URL.
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local-dev-key")
resp = client.chat.completions.create(
model="gpt-oss-20b",
messages=[{"role": "user", "content": "Give me one sentence on why GGUF matters."}],
max_tokens=512,
)
print(resp.choices[0].message.content)
Step 7: Tune the flags that matter
Most tuning comes down to a short list of switches. Run llama-server –help against your own build before relying on any of them, since names and defaults shift between releases.
| Flag | What it does | Guidance |
|---|---|---|
| -m or -hf | Loads a local file or downloads from Hugging Face | Use -m for pinned files, -hf for convenience |
| -c | Total context window in tokens | Start small, raise only if you need it |
| -ngl | Layers to offload to the GPU | 99 means everything, lower it when VRAM runs out |
| –parallel or -np | Concurrent request slots | Context is divided across slots |
| –jinja | Applies the model’s embedded chat template | Keep it on for modern chat and reasoning models |
| –cache-type-k and –cache-type-v | KV cache precision | q8_0 roughly halves cache memory versus f16 |
| –n-cpu-moe | Keeps expert tensors of the first N layers on the CPU | Lets a large mixture-of-experts model fit a small GPU |
| -t | CPU threads | Match physical cores for CPU inference |
| –api-key | Requires a bearer token | Mandatory the moment you bind beyond localhost |
| –host and –port | Where the server listens | Always set both explicitly |
When VRAM is tight, work through the list in order. Cut the context first, then quantize the KV cache, then move expert layers to the CPU, and only then drop to a smaller quant. Here is a launch line for a 16 GB card that wants more headroom:
llama-server -m "$MODEL" --alias gpt-oss-20b --host 127.0.0.1 --port 8080 \
-c 8192 -ngl 99 --n-cpu-moe 8 \
--cache-type-k q8_0 --cache-type-v q8_0 --jinja
A quantized V cache depends on flash attention, which recent builds enable automatically on supported backends. If the server refuses to start with a quantized V cache, drop the V setting or check –help for how your build toggles flash attention. Change one flag at a time and watch the log. The offload summary and the memory lines tell you immediately whether a change helped.
Steps 8 and 9: Run Open WebUI and Connect It to llama-server #
Step 8: Start Open WebUI in Docker
Open WebUI is a self-hosted chat interface that talks to any OpenAI-compatible endpoint. It is the interface layer only. It doesn’t quantize models and it doesn’t run inference, which is exactly why it pairs well with llama-server. For this standalone test, make sure llama-server is reachable from the container, protect it with an API key, and keep the port closed to the outside network with your firewall.
llama-server -m "$MODEL" --alias gpt-oss-20b \
--host 0.0.0.0 --port 8080 \
-c 16384 -ngl 99 --jinja --parallel 2 \
--api-key local-dev-key
docker run -d --name open-webui --restart unless-stopped \
-p 127.0.0.1:3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OPENAI_API_BASE_URL=http://host.docker.internal:8080/v1 \
-e OPENAI_API_KEY=local-dev-key \
-e ENABLE_OLLAMA_API=false \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:main
The :main tag floats. Once the setup works, swap it for a specific version tag so an unattended restart can’t pull a new release into production. The named volume keeps accounts, chats and settings across upgrades. Prefer to skip Docker? On Python 3.11, pip install open-webui gives you the same interface.
Open http://localhost:3000. The first account you create becomes the administrator, so make it yours before anyone else on the network finds the page.
Step 9: Connect, verify and lock down
Go to Admin Panel, then Settings, then Connections. You should see the OpenAI-style connection that your environment variables created, pointing at the /v1 address. The trailing /v1 matters. Without it the model list comes back empty. Click the verify button beside the URL. A green confirmation means Open WebUI reached the server and the key matched. Start a new chat, pick gpt-oss-20b from the model menu and send a prompt. The server console should print the request as it streams.
Then close the door. Under Admin Panel and Settings, then General, switch off new sign-ups once your team has accounts. Settings created from environment variables are stored in Open WebUI’s database after the first launch. Changing the variable later does nothing, so edit them in the admin panel, or wipe the volume if you really want to start over. Tune per-model defaults in the same area: temperature, a system prompt, and context length hints.
Two housekeeping notes. Secondary reports say Open WebUI is free to self-host but ships under its own license with a branding clause, which matters only if you plan to rebrand or resell it, and HokAI’s licence and limits write-up walks through the details. Read the license text yourself before white-labeling. Also remember that the interface can be pointed at several backends. A team can keep llama-server for private work and add a hosted API for hard tasks, as discussed earlier, and a coding agent such as Kilo Code in VS Code can share the same llama-server endpoint with the browser chat.
Step 10: Convert and Quantize Your Own GGUF #
Sooner or later the model you want ships as safetensors with no GGUF, or the published quant isn’t the size you need. Converting takes three commands. To keep the exercise fast, this walkthrough uses a tiny model from the Qwen3 family. Swap in any architecture that llama.cpp supports, and check its license first.
cd llama.cpp
pip install -r requirements.txt
mkdir -p ../hf ../out
hf download Qwen/Qwen3-0.6B --local-dir ../hf/qwen3-0.6b
python3 convert_hf_to_gguf.py ../hf/qwen3-0.6b \
--outfile ../out/qwen3-0.6b-f16.gguf --outtype f16
llama-quantize ../out/qwen3-0.6b-f16.gguf ../out/qwen3-0.6b-Q8_0.gguf Q8_0
llama-quantize ../out/qwen3-0.6b-f16.gguf ../out/qwen3-0.6b-Q4_K_M.gguf Q4_K_M
llama_model_quantize_impl: model size = N.NN MiB
llama_model_quantize_impl: quant size = N.NN MiB
main: quantize time = N ms
The F16 file is your master copy. Always quantize from it, never from another quant. For the lower-bit types, build an importance matrix from a calibration text. Any plain-text file of a few thousand tokens that resembles your real prompts will do. Code-heavy workloads deserve code in the calibration set.
llama-imatrix -m ../out/qwen3-0.6b-f16.gguf -f calibration.txt \
-o ../out/imatrix.dat -ngl 99
llama-quantize --imatrix ../out/imatrix.dat \
../out/qwen3-0.6b-f16.gguf ../out/qwen3-0.6b-IQ4_XS.gguf IQ4_XS
Test the result the same way you tested the download. Run llama-cli with one of your new files and confirm it answers sensibly. A converter error that says the architecture is unknown means your llama.cpp checkout predates the model. Pull the latest code and rebuild before blaming the weights. Converting a large model needs memory for the full-precision tensors, so a 30B conversion on a 16 GB laptop will crawl or fail. Do big conversions on a machine with ample RAM, or download an existing quant instead.
Step 11: Benchmark Speed and Quality Across Quants #
Pick a quant with measurements, not hunches. llama-bench reports speed, and llama-perplexity reports how far a quant drifts from the original. Run both on every file you consider.
for f in ../out/qwen3-0.6b-Q8_0.gguf ../out/qwen3-0.6b-Q4_K_M.gguf ../out/qwen3-0.6b-IQ4_XS.gguf; do
llama-bench -m "$f" -p 512 -n 128 -ngl 99
done
bash scripts/get-wikitext-2.sh
llama-perplexity -m ../out/qwen3-0.6b-f16.gguf -f wikitext-2-raw/wiki.test.raw -ngl 99
llama-perplexity -m ../out/qwen3-0.6b-Q4_K_M.gguf -f wikitext-2-raw/wiki.test.raw -ngl 99
| model | size | params | backend | ngl | test | t/s |
| ------------------------ | -------: | -------: | ------- | --: | ----: | -----------: |
| qwen3 0.6B Q4_K - Medium | N.NN MiB | N.NN B | CUDA | 99 | pp512 | NNNN.NN +- NN |
| qwen3 0.6B Q4_K - Medium | N.NN MiB | N.NN B | CUDA | 99 | tg128 | NNN.NN +- NN |
Final estimate: PPL = N.NNNN +/- 0.0NNNN
The numbers above are placeholders. Your hardware sets the real values, which is the point of measuring. Read the table like this:
| Column or test | Meaning | What to do with it |
|---|---|---|
| pp512 | Prompt processing speed on a 512-token prompt, in tokens per second | Drives wait time for long documents and big system prompts |
| tg128 | Text generation speed over 128 new tokens | Drives how fast the answer streams to the reader |
| size | File size in memory | Compare against your memory budget from Step 1 |
| PPL | Perplexity on the test text, lower is better | Compare each quant against the F16 baseline |
Expect Q8_0 to land almost on top of F16, Q4_K_M a little higher, and anything below four bits to climb quickly. Generation speed usually rises as files shrink, because decoding is limited by memory bandwidth. Choose the smallest quant whose perplexity gap you can live with, then confirm with your own prompts. Perplexity is a rough proxy. A coding assistant that fails on structured output matters more than a small shift in a wikitext score.
Step 12: The Complete Working Project, Packaged With Docker Compose #
Everything so far ran by hand. This final step freezes it into a project you can copy to another machine and start with one command. The layout is small:
local-llm-stack/
.env
compose.yaml
models/
scripts/
smoke_test.py
bench.py
Start with the environment file. Generate real secrets with openssl rand -hex 32 and keep the file out of version control.
MODEL_REPO=ggml-org/gpt-oss-20b-GGUF
MODEL_ALIAS=gpt-oss-20b
CTX=16384
LLAMA_API_KEY=replace-with-a-long-random-string
WEBUI_SECRET_KEY=replace-with-another-long-random-string
HF_TOKEN=
The Compose file runs two services on a private network. llama-server isn’t exposed to the LAN at all. Only Open WebUI is published, and only on the loopback address, so a reverse proxy decides who gets in. The health check keeps Open WebUI from starting before the model finishes .
services:
llama:
image: ghcr.io/ggml-org/llama.cpp:server-cuda
restart: unless-stopped
environment:
LLAMA_CACHE: /models
HF_TOKEN: ${HF_TOKEN:-}
volumes:
- ./models:/models
ports:
- "127.0.0.1:8080:8080"
command: >
-hf ${MODEL_REPO}
--alias ${MODEL_ALIAS}
--host 0.0.0.0 --port 8080
-c ${CTX} -ngl 99
--jinja --parallel 2 --metrics
--api-key ${LLAMA_API_KEY}
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
interval: 15s
timeout: 5s
retries: 40
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
webui:
image: ghcr.io/open-webui/open-webui:main
restart: unless-stopped
depends_on:
llama:
condition: service_healthy
ports:
- "127.0.0.1:3000:8080"
environment:
OPENAI_API_BASE_URL: http://llama:8080/v1
OPENAI_API_KEY: ${LLAMA_API_KEY}
ENABLE_OLLAMA_API: "false"
WEBUI_SECRET_KEY: ${WEBUI_SECRET_KEY}
volumes:
- open-webui:/app/backend/data
volumes:
open-webui:
docker run --rm --gpus all ubuntu nvidia-smi # confirm GPU access first
docker compose up -d
docker compose logs -f llama
On a Mac, skip the llama container. Docker on macOS can’t reach Metal, so run llama-server natively and keep only Open WebUI in Compose, pointing at host.docker.internal. On a CPU-only box, use the plain server image and drop the deploy block.
Next, two scripts that verify the stack and measure it. The smoke test uses only the standard library, so it runs anywhere.
import json
import os
import sys
import urllib.request
BASE = os.environ.get("LLM_BASE", "http://127.0.0.1:8080")
KEY = os.environ.get("LLAMA_API_KEY", "")
def call(path, payload=None):
data = json.dumps(payload).encode() if payload else None
req = urllib.request.Request(
BASE + path,
data=data,
headers={"Content-Type": "application/json", "Authorization": "Bearer " + KEY},
)
with urllib.request.urlopen(req, timeout=120) as resp:
return json.load(resp)
def main():
health = call("/health")
assert health.get("status") == "ok", health
ids = [m["id"] for m in call("/v1/models")["data"]]
print("models:", ids)
reply = call("/v1/chat/completions", {
"model": ids[0],
"messages": [{"role": "user", "content": "Reply with the single word: ready"}],
"max_tokens": 256,
})
text = (reply["choices"][0]["message"].get("content") or "").strip()
print("reply:", text)
if not text:
print("empty content: raise max_tokens, reasoning may have used the budget")
sys.exit(1)
if __name__ == "__main__":
main()
python
import json
import os
import statistics
import urllib.request
BASE = os.environ.get("LLM_BASE", "http://127.0.0.1:8080")
KEY = os.environ.get("LLAMA_API_KEY", "")
MODEL = os.environ.get("MODEL_ALIAS", "gpt-oss-20b")
PROMPT = "Write a 200-word explanation of how a hash table handles collisions."
RUNS = 5
def one_run():
body = json.dumps({
"model": MODEL,
"messages": [{"role": "user", "content": PROMPT}],
"max_tokens": 400,
"temperature": 0,
}).encode()
req = urllib.request.Request(
BASE + "/v1/chat/completions",
data=body,
headers={"Content-Type": "application/json", "Authorization": "Bearer " + KEY},
)
with urllib.request.urlopen(req, timeout=300) as resp:
timings = json.load(resp).get("timings", {})
return timings.get("prompt_per_second"), timings.get("predicted_per_second")
def main():
one_run() # warm-up, loads caches
prompt_speeds, gen_speeds = [], []
for _ in range(RUNS):
pp, tg = one_run()
if pp and tg:
prompt_speeds.append(pp)
gen_speeds.append(tg)
if not gen_speeds:
print("no timings returned: check that the server build reports them")
return
print("runs:", len(gen_speeds))
print("prompt tokens/s (median):", round(statistics.median(prompt_speeds), 1))
print("generation tokens/s (median):", round(statistics.median(gen_speeds), 1))
if __name__ == "__main__":
main()
export LLAMA_API_KEY=replace-with-a-long-random-string
python3 scripts/smoke_test.py
python3 scripts/bench.py
models: ['gpt-oss-20b']
reply: ready
runs: 5
prompt tokens/s (median): XXX.X
generation tokens/s (median): XX.X
Run the smoke test after every image update. If it passes, the server, the key and the chat template all work together. For machines without Docker, a systemd unit keeps a native llama-server alive across reboots:
[Unit]
Description=llama.cpp server
After=network-online.target
[Service]
User=llm
ExecStart=/opt/llama.cpp/build/bin/llama-server \
-m /srv/models/gpt-oss-20b.gguf --alias gpt-oss-20b \
--host 127.0.0.1 --port 8080 -c 16384 -ngl 99 --jinja
Restart=on-failure
[Install]
WantedBy=multi-user.target
Put a reverse proxy in front of Open WebUI to handle TLS, pointing it at the local port where Open WebUI listens. Teams that mix local and hosted models can route the hosted half through the Cloudflare AI Gateway for logging and spending caps, while the local half stays on this stack.
Common Pitfalls When You Run an LLM Locally #
Most failed setups trace back to a handful of avoidable mistakes. These eight cause the bulk of the support threads.
- Running a CPU-only build and not noticing. The server works, the answers arrive, and everything is ten times slower than it should be. Check the offload line in the startup log every time you change builds or images.
- Leaving out –jinja. Without the embedded chat template, some models produce broken turns, ignore system prompts or fail at tool calls. Modern GGUF files carry the template, so use it.
- Pointing a container at localhost. Inside a container, localhost means the container. Use the service name in Compose or host.docker.internal for a server on the host, and bind the server to an address the bridge can reach.
- Assuming environment variables always apply. Open WebUI stores many settings in its database after first launch. Edit connections and sign-up rules in the admin panel, not in the Compose file.
- Exposing the server without a key. Letting the server listen on the whole network with no –api-key hands your GPU to anyone on it. Set the key and firewall the port even on a home LAN.
- Trusting the default port. Reports say a recent llama.cpp release changed the server default to 9931. Whatever your build does, pass –port explicitly and your clients never break on upgrade.
- Requantizing a quantized file. Each round adds error. Quantize from F16 or BF16 only, and don’t touch models that ship in a native compact format.
One more habit pays off: keep a text file listing the llama.cpp build, the model file name, the flags and the Open WebUI tag that worked. When something breaks after an update, that file turns a mystery into a diff.
Troubleshooting: Fixes for the Errors You Will Actually See #
Server won’t start or crashes
- CUDA out of memory at load. The weights plus KV cache exceed VRAM. Lower -c, quantize the KV cache with q8_0, move experts to the CPU with –n-cpu-moe, or pick a smaller quant. Close other GPU programs too, since browsers and games hold VRAM.
- Out of memory only after a long chat. The KV cache grows with the conversation. Reduce -c or –parallel, or restart the server to clear fragmented slots.
- Unknown model architecture. Your build is older than the model. Update llama.cpp, rebuild, and retry. If the architecture still fails, it may not be supported yet, so check the project’s release notes.
- Address already in use. Another process holds the port, often an earlier llama-server. Find it with ss -ltnp on Linux or Get-NetTCPConnection on Windows, stop it, or choose another –port.
- Download errors from Hugging Face. If the model is gated, accept its terms on the model page, create a read token, and run hf auth login or pass HF_TOKEN into the container.
- Windows says a CUDA DLL is missing. The prebuilt CUDA binary needs the matching CUDA runtime files next to it. Download the runtime bundle from the same release page and copy the DLLs beside llama-server.exe.
Open WebUI can’t see the model
- Model list is empty. The connection URL is probably missing the /v1 suffix, or the key doesn’t match the one passed to –api-key. Fix both in Admin Panel, Settings, Connections, then use the verify button.
- Connection refused from the container. The server is listening only on the host’s loopback interface, and the container arrives over a bridge network. Make the server listen on the network with a key and a firewall rule, or run both services in Compose where they share a network.
- 404 on /v1 routes. You are talking to the wrong port or an old server build. Hit /health and /props directly with curl to see what is listening.
- Compose starts Open WebUI before the model loads. A large model takes minutes to load. Keep the health check and depends_on condition from Step 12, and raise the retries if your disk is slow.
Slow, empty or garbled output
- Generation crawls. Look for fewer offloaded layers than the model has. Raise -ngl, free VRAM, or use a smaller model. On CPU-only boxes, set -t to your physical core count rather than the thread count.
- Empty replies from reasoning models. The model spent the whole max_tokens budget on hidden reasoning. Raise max_tokens substantially, or lower the reasoning effort through the chat template options if your model supports it.
- Garbled or repeating text. Usually a wrong or missing chat template, or a context window pushed far past what the model supports. Add –jinja, shrink -c, and make sure you are not mixing a base model with an instruct template.
- A Mac runs slower than expected. Apple silicon caps how much unified memory the GPU may wire. Close memory-hungry apps first. On recent macOS versions you can raise the GPU wired limit with a sysctl setting, but do it carefully and read Apple’s documentation for your release.
Advanced Tips for Faster, Safer Local Inference #
Offload experts, not layers. Mixture-of-experts models activate a small slice of their weights per token, so the expert tensors can sit in system RAM while attention layers stay on the GPU. The –n-cpu-moe flag does this layer by layer. Reports on the v0.7.0 release also mention a host-memory expert cache, but check your own build’s –help output before relying on it.
Try speculative decoding. A small draft model proposes several tokens and the big model verifies them in one pass. When both share a tokenizer, throughput can rise noticeably with no change in output. The -md flag loads the draft, and –draft-max caps how many tokens it proposes. Benchmark with Step 12’s script, since gains depend on the prompt type.
Constrain the output. llama-server accepts JSON schemas and grammars, which forces valid structured output even from small models. For extraction pipelines this beats retry loops. Pair it with a low temperature and a short max_tokens.
Serve embeddings from the same box. Start a second llama-server instance with –embeddings and an embedding model, then give Open WebUI that endpoint for document search. Keeping embeddings local keeps the whole retrieval path private.
Swap models on demand. One llama-server process serves one model. Community proxies such as llama-swap start and stop instances based on the model name in each request, so a single endpoint can front several GGUF files without holding them all in memory at once.
Watch it in production. The –metrics flag publishes Prometheus counters for prompt and generation throughput and slot use. Scrape them, alert on queue depth, and you will see saturation before users complain.
Pin everything. Fix the llama.cpp image tag, the Open WebUI version, and the model file hash. Upgrade on a schedule, run the smoke test, and roll back if it fails. New models arrive weekly, and our AI coverage hub is a good place to see which ones deserve a test slot.
FAQ: Running LLMs Locally With llama.cpp and Open WebUI #
Is llama.cpp better than Ollama?
Neither wins outright. llama.cpp gives you direct control over every flag and the freshest model support, and Ollama gives you a simpler install and a model registry. Ollama bundles a pinned llama.cpp build, so you can think of llama.cpp as the engine and Ollama as one way to package it.
How much memory do I need to run an LLM locally?
Use the weights formula: parameters in billions times bits per weight, divided by eight, then add the KV cache and about 10 percent headroom. OpenAI quotes roughly 16 GB for gpt-oss-20b and roughly 80 GB for gpt-oss-120b. Small 4B to 8B models at 4-bit fit in 8 GB.
Which quantization should I choose?
Start with Q4_K_M for general use and Q8_0 when memory is plentiful. Models published in MXFP4, such as gpt-oss, should run as shipped. Measure perplexity and your own prompts before settling, as Step 11 shows.
Can I run the newest Chinese open-weight models locally?
Only when weights, llama.cpp support and a GGUF all exist. In the sources I checked, Xiaomi says MiMo-V2.6 is open-sourced but no official GGUF is confirmed, GLM-5.3-Flash’s weight status is disputed, and GLM-Legal is a gated API. Run the checklist table before down anything large.
Is Open WebUI free?
Third-party reports describe the self-hosted software as free, with sales-led enterprise licensing for extra features and a branding clause in its custom license. Infrastructure and any paid model APIs you connect cost extra. Confirm the terms on the project’s own pages before commercial use.
Does running locally guarantee privacy?
No. The model stays on your machine, but any plugin, web-search tool or remote tool server you attach can send data out. Review each integration, bind services to private addresses, and require keys.
How do I update llama.cpp without breaking my setup?
Pin the image tag or build number, upgrade on a test copy, run the smoke test and the benchmark script, and compare results with your notes. Roll back by restoring the previous tag if anything regresses.