Running LLMs Without a GPU: What's Actually Possible

The honest numbers on CPU, NPU, and iGPU inference — what runs, how fast, and when to stop pretending.

Last year a friend who runs a small consulting firm in Dubai asked me what GPU he needed to "run AI locally." He had a budget in mind and had already been quoted a four-figure price tag for a workstation. I asked him what he actually wanted to run. A document chatbot over his firm's contracts. He owned a three-year-old office laptop with 16 GB of RAM and no discrete GPU.

I told him not to buy anything yet. Two hours later, I had a 7B-parameter model running on that laptop at about 11 tokens per second, answering questions from his own PDFs. Not fast enough for a chat product serving a thousand users. Fast enough for a private assistant that costs nothing per query and keeps every contract on his machine.

The market wants you to believe local AI requires expensive hardware. The truth is more interesting and much cheaper. Here is what is actually possible — with real numbers, real architecture, and an honest map of where CPU-only inference stops being viable.

The confusion comes from a conflation of two different workloads. Training and fine-tuning a model is brutally compute-hungry: it means doing forward passes over billions of parameters millions of times, and backpropagation on top. That genuinely wants GPUs, and a lot of them.

Inference — running the model to produce an answer — is a different animal. Generating one token means one forward pass through the network. It is memory-bandwidth bound far more than compute bound, especially at the small batch sizes a single user generates. That is the single most important fact in this entire article: for single-user inference, what matters is how fast you can stream weights from memory, not how many FLOPS you have. A CPU with lots of fast RAM can do a surprisingly respectable job, and modern processors have neural accelerators that help even more.

Here are the real token rates I have measured on actual hardware, not the marketing ones. Your mileage varies, but these are the right ballparks for current consumer machines:

For context: comfortable reading speed is roughly 20 tokens per second. Below 8 tokens per second, interactive chat starts to feel sluggish, though a batch job — summarize these 400 documents overnight — does not care about interactivity at all, which changes the calculus completely.

One more distinction that explains why CPU inference feels uneven in practice: the two phases of generation behave very differently. Prompt processing (reading the input, aka prefill) is compute-bound and can be slow on CPU for long inputs — a 2,000-token prompt might take a second or two before the first output token appears. Token generation (decode) is memory-bandwidth-bound and steady — once the model is warmed up, each output token streams at a consistent rate. So the experienced quality of a CPU model is dominated by prompt length: short prompts with long answers feel fine; long prompts with short answers feel sluggish, and that is pure prefill time, not generation speed. Capping context and keeping inputs tight is not a quality compromise, it is a latency lever.

The magic ingredient on the Apple side is unified memory: the same memory serves CPU and GPU, and it is fast memory. That is why a fanless MacBook Air outperforms a bigger Windows laptop for llama.cpp. On the Windows/Intel side, the NPUs shipping in 2024+ laptops are the emerging story — they are built for low-power token generation, and software support is maturing quickly.

Every serious CPU inference setup is built from the same four pieces:

1. Quantized models (GGUF). The breakthrough that made CPU inference practical. You take a model whose weights are 16-bit floats and shrink them to 4-bit integers, trading a little quality for a ~4x memory and bandwidth reduction. The GGUF format (from the llama.cpp project) is the standard container. The Q4_K_M variant is the sweet spot I default to — decent quality, ~5 GB for an 8B model, fits in a laptop's budget. Q8_0 is better quality at ~9 GB; Q3_K_M fits in 4 GB but you will feel the quality loss.

2. llama.cpp. The reference runtime. A pure C/C++ implementation that runs on CPU, and offloads layers to GPU/NPU where available. It is the engine behind virtually every local-LLM tool you have heard of, and it is fast on CPU because it was written to be.

3. A server layer. llama.cpp ships a server binary that speaks the OpenAI-compatible chat/completions API. This is the detail that makes CPU inference actually usable: any tool that talks to the OpenAI API — including your existing app, if you point its base_url

at your local server — will work against your CPU model with a one-line change.

4. Ollama (or llama-cpp-python) for the glue. Ollama wraps llama.cpp with model management and a dead-simple CLI; llama-cpp-python

gives you the same engine as a Python package. Both are legitimate; you are choosing convenience over control.

Here is the whole setup on a Mac or Linux box. First, install Ollama and pull a quantized model:

ollama pull llama3.1:8b-q4_K_M
ollama serve

That single command is now a running LLM endpoint on http://localhost:11434

, OpenAI-compatible. Test it:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.1:8b-q4_K_M",
       "messages": [{"role": "user", "content": "Explain quantization in 50 words."}]}'

If you prefer raw llama.cpp for control — say, you want to offload specific layers to an iGPU or NPU — you clone it and run the server directly:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build --config Release
./build/bin/llama-server \
  -m llama-3.1-8b-instruct-Q4_K_M.gguf \
  --n-gpu-layers 12 \
  -c 4096

The --n-gpu-layers

flag is the whole game on hybrid hardware: you push the layers that benefit from the GPU/NPU and keep the rest on CPU, trading power draw for speed. On the Apple side the equivalent is Metal layers (-ngl

with Metal enabled).

Now the part people forget. Point your OpenAI-compatible client at it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
    model="llama3.1:8b-q4_K_M",
    messages=[{"role": "user", "content": "Summarize the attached contract risk clauses."}],
)
print(resp.choices[0].message.content)

That is it. Your app now runs on a local model with zero per-query cost and zero data leaving the machine. The entire "I need a GPU" story collapses into one base_url

change.

If you do not know which model to start with, here is the shortlist I use. For a first experiment, a 7–8B model at Q4_K_M (Llama 3.1, Qwen 2.5, Gemma) is the right default on 16 GB of RAM — big enough to feel genuinely capable, small enough to be usable. If you are on 8 GB or an older machine, step down to a 3–4B model at Q4_K_M; the quality gap is smaller than the speed gap suggests. And if you only need classification or extraction, a 1–3B model at 40+ tok/s will beat an 8B at 10 tok/s for the job. The rule: pick the smallest model that passes your acceptance test, then measure — most people overestimate what they need by one size.

If you are on CPU-only and the speed is not acceptable, the fix is almost never "get a GPU." Work through this order:

-c

from 8192 to 4096 makes the first token arrive much faster.And here is the counterintuitive one: for batch work, a slow model is a non-issue. If you need to tag 10,000 support tickets overnight, 5 tok/s is plenty. You are optimizing for throughput over time, not first-token latency, and CPU inference wins that comparison on cost by a mile.

The same logic extends to embeddings, which people forget entirely. An embedding model like a small 300M–1B parameter encoder runs happily on CPU and is the backbone of local RAG: it turns your documents into vectors on your own machine, with no API calls and no data leaving the building. I have run a full local retrieval pipeline — embeddings plus a small chat model — on a single 16 GB laptop, and the retrieval step, which is the part that actually makes answers good, is nearly instant. If your goal is a private document assistant, the LLM is the part you have to accept as "good enough"; the embeddings are the part that genuinely shines.

I want to be equally honest about the wall. CPU-only inference fails in four specific places, and you should not pretend otherwise:

You need the GPU when the workload is compute-bound or concurrent: fine-tuning, large-scale embedding jobs, or serving many simultaneous users. If you are building the next consumer chatbot product, you are not the CPU story. You are the cloud API story.

But here is the framing I use now, after years of building: the GPU question is not "should I buy one" — it is "what is the workload, what is the concurrency, and what are the latency requirements?" A private document assistant, a code-completion tool running on a single developer's machine, an on-prem compliance filter that must never see the internet, a batch tagging pipeline — all of these are CPU-viable today, and the hardware you already own is probably enough to start.

When someone asks me to stand up local LLMs on the hardware they have, I work through this list:

base_url

changeMy friend in Dubai runs his contract assistant on that same laptop today. It does not write poetry and it is not answering a thousand concurrent users. It reads contracts, answers questions in context, and costs him nothing per query — and his documents never leave his machine. That is the actual value proposition of CPU inference: not "local models beat the cloud," but "local models are good enough for a specific class of work, and that class is much bigger than people think."

Start with what you already own. Quantize, cap context, offload what you can, and match the model to the job. You will be surprised how far a laptop goes — and you will only spend GPU money when the workload genuinely demands it.

*Gulshan Yad