Part 1 established the hardware, the runner, and the primary model. This entry covers what governs inference on that hardware — the two phases of inference, the cost of long context, and the cost of a model from disk — and compares the local machines against a free-tier cloud model.
Inference has two phases. Prefill processes the input prompt before any output appears; it is compute-bound and wants a GPU. Generation produces output tokens one at a time and is bound by memory bandwidth. Casual use is almost all generation and hides the difference; the cost of prefill surfaces only when prompts grow large.
| Machine | CPU / RAM | GPU (VRAM) | Storage (read) | Prefill (tok/s) | Gen (tok/s) | Load (18 GB) |
|---|---|---|---|---|---|---|
| Primary desktop | 5950X / ~80 GB DDR4 | RX 6900XT (16 GB) | NVMe (~2.1 GB/s) | 360 | 18.3 | 8.4s | | Secondary box | 5600G / 32 GB DDR4 | GTX 1060 (6 GB) | SATA SSD (~0.35 GB/s) | 253 | 17.1 | 50.6s | | Laptop | 8840U / 32 GB DDR5 | Radeon 780M (none) | NVMe (~2.4 GB/s) | 20 | 10.0 | 7.5s |
All inference figures come from a controlled run: the same model (Gemma 4 26B, 18 GB) on each machine, a unique random prefix per prompt to defeat caching, a fixed 8,192-token context, warm, on an identical ~6,855-token prompt (generation timed over a 200-token output).
Two things stand out. Prefill varies about eighteen-fold across the machines (360 to 20 tok/s) while generation varies less than twofold (18.3 to 10.0), and prefill is what dominates large-prompt workloads — so a machine can look fine on generation yet be useless in practice. Model-load time, separately, is set by storage rather than compute: the secondary box's budget SATA SSDs load the 18 GB model in 50 seconds against eight on NVMe, which turns a cold request into a minute-long stall.
| Secondary box | Request time |
|---|---|
| Warm (model resident) | ~4s |
| Cold (model reload) | ~54s |
If the model is allowed to unload between calls, every call silently pays that reload — a real source of intermittent timeouts. The fix is a long keep-alive (`OLLAMA_KEEP_ALIVE=24h`
) with pre-warming; on a slow-disk node it is a precondition, not a refinement.
The laptop deserves particular attention, because it is sold as an "AI PC," and that framing is precisely what it fails to honour. The 8840U (AMD's 8040 "Hawk Point" series) carries a dedicated XDNA NPU rated at up to 16 TOPS — around 38 across the platform — and is marketed under the "Ryzen AI" banner for exactly this sort of local inference. Yet the NPU is built for low-power, always-on tasks such as webcam background effects and noise suppression, and the LLM runner does not address it at all. Large-model inference therefore falls to the CPU, which prefills at only ~20 tok/s (the table above), so a system prompt of ten to fifteen thousand tokens needs eight to twelve minutes to ingest before a single token is produced — the thirteen-minute stall observed in practice.
The lesson cuts against the marketing twice over. "AI PC" denotes a narrow class of accelerated workloads that excludes running a multi-billion-parameter model against a large prompt; the advertised TOPS are, for this purpose, inert, and the figure that decided the outcome was an unglamorous CPU prefill rate. The same NPU also sits below the 40-TOPS threshold Microsoft attaches to the AI-PC label.
Long context is paid for in memory, because the KV cache grows linearly with context length. The runner defaults to a 4K–8K window; this was raised to 64K through a custom Modelfile (num_ctx 65536
).
| Context (q8_0 KV cache) | KV cache size | Verdict |
|---|---|---|
| 64K | ~926 MiB GPU + 231 MiB CPU | stable — adopted |
| 128K | larger; slower prefill, unstable | rejected |
The 64K window was kept as the stable operating point. (Forcing all layers onto the 16 GB card with num_gpu 99
fails outright; rely on the runner's automatic GPU/CPU split instead.)
How do the local machines compare with a hosted model? On the same ~150-word reasoning prompt, a free-tier cloud model (Gemini 3 Flash) was timed end-to-end against the two local GPUs.
| Option | End-to-end latency | Output |
|---|---|---|
| Cloud — Gemini 3 Flash (free tier) | ~5.8s | |
| 205 tokens, after ~536 internal reasoning tokens | ||
| Primary desktop — RX 6900XT | ~20.6s | 200 tokens |
| Secondary box — GTX 1060 | ~64.5s | 200 tokens (inflated by a cold reload) |
This is not a pure-compute comparison — the cloud figure includes the network round-trip and Google's serving infrastructure, and the API exposes no prefill/generation split — but it measures the quantity that matters in use: how quickly an answer arrives. The cloud model won comfortably while doing more work, spending ~536 internal reasoning tokens before its 205-token answer. The lesson is not that cloud beats local, but that placement should follow the task: routine, high-volume work belongs local — private, unmetered, and predictable in latency.
These findings — prefill needs a GPU, context costs memory linearly, a cold model is far dearer than a resident one, and placement should follow the task — compound once more than one model must be held in memory at once. Whether several models can coexist without contention is the subject of the next entry.