Local Inference Guide
A deep, hands-on guide to local inference for programmers who know how to code but have never touched AI tooling. Every term is defined before it's used, and every tutorial section has commands you can actually run.
What is "local inference"? Running a language model on hardware you control — laptop, workstation, or home server — instead of calling a hosted API. See
Concepts & Glossaryfor the full breakdown.
Why run models locally #
No API costs or rate limits— you pay in electricity and hardware, not per-token billing.** Privacy**— nothing leaves your machine.** Offline capability**— works with no network at all.** Control**— choose the exact model, quantization, and sampling behavior.** Low latency**— no network round-trip if embedded in a local app.
Where to start #
New to all of this? Read top to bottom. Already know the vocabulary? Jump straight to a tutorial. Concepts & Glossary Start here
Tokens, weights, quantization, KV cache, context window — every term defined once, clearly.
Sizing Your Hardware
The formula for figuring out what model size fits your GPU or RAM.
Choosing a Tool
Ollama vs. llama.cpp vs. vLLM vs. transformers — what each one is actually for.
Ollama Tutorial 1
The fastest path from zero to a running local model.
llama.cpp Tutorial 2
Build from source, quantize manually, run the raw inference engine.
Python / transformers Tutorial 3
Full programmatic control using Hugging Face's library.
vLLM Tutorial 4
High-throughput serving for many concurrent users.
Building an App Tutorial 5
A portable streaming client that works against any of the above.
Choosing a Model
Qwen3, Gemma, Llama, DeepSeek, GLM — what's worth running in 2026.
Performance & Troubleshooting
GPU off, quantization tradeoffs, and fixes for common errors.
Forum
Ask questions and discuss local inference with other readers.