How much intelligence can you run locally?

wpnews.pro

Key findings #

57.6 vs 34Claude Opus 5.5 leads the AA Index v4.3.2 at 57.6. The best model that fits one GPU scores about 34, and the best open-weight model, Xiaomi's MiMo-V2.6-Pro, scores 46.3. Intelligence gap
4 monthsEpoch AI puts the lag between the best open and closed models at 4 months. An earlier Epoch analysis found that models small enough for one consumer GPU trail by 6–12 months. Intelligence gap

- 3.5×The RTX 5090 (1,792 GB/s) finished Artificial Analysis's local agent workload more than 3.5× faster than the 128 GB unified-memory machines (256–307 GB/s). [Decode speed](#speed)
- 2.5–3.3×Qwen3.6-35B-A3B, a mixture-of-experts model with 3B active parameters, ran 2.5–3.3× faster than the dense Qwen3.8-27B on every system AA tested. [Decode speed](#speed)

≤1%4-bit weight-only quantization cost reasoning models 1% or less. Below 4 bits the losses grow quickly, most of all for small models and hard tasks. Quantization
$5,000A new RTX 5090 (MSRP $1,999) costs about $5,000 in North America. The 128 GB Framework Desktop went from $1,999 to $3,449. Hardware and prices
11.7 → 50On one DGX Spark, gpt-oss-120b decodes at 11.7 tok/s in Ollama, 38.5 in tuned llama.cpp and about 50 in SGLang. Inference engines

Artificial Analysis Intelligence Index v4.3.2 · Epoch AI

Intelligence gap #

The AA Intelligence Index is the most useful single scale for this comparison, as long as all scores come from the same version. It moved from v4.1 in June to v4.3.2 in October, and scores from different versions are not comparable.

The open models that come closest to the frontier are too large for a desk. Kimi K3 has 2.8 trillion parameters with 104 billion active, and DeepSeek V4.1 Flash has 552 billion, which fits a 512 GB Mac Studio at best. Among models that do fit, newer beats bigger: Qwen3.8-27B outscores every 120B-class model that fits on a 128 GB machine by more than 20 points, except Qwen3.8-Flash-Next.1

Epoch AI estimates that the best open-weight models trail the best closed models by about 4 months (Edwards & Emberson, May 29, 2026). An earlier Epoch analysis put the lag for models that fit one consumer GPU at 6–12 months (Somala & Emberson, Aug 2025), and notes that small open models are more likely to be tuned for specific benchmarks, so their real-world lag may be longer.

Open models also cost much less through a hosted API. At the v4.1 launch in June, AA found DeepSeek V4 Pro cost $0.04 per Index task, against $1.78 for Claude Opus 4.8.24

Measured tokens per second · AA-AgentPerf-Local

Decode speed #

Generating a token means reading every active weight from memory once. There is little arithmetic per byte, so the chip spends most of its time waiting on memory, and speed is capped at roughly memory bandwidth divided by the bytes read per token. On any model that fits in its 32 GB, an RTX 5090 at 1,792 GB/s is much faster than a 128 GB machine at 273 GB/s.

With gpt-oss-120b selected, the RTX 5090 drops out because the model needs about 66 GB. The 128 GB machines run it at about 55–88 tok/s, since each token reads only about 5 GB of the 66. On a dense 70B model, where every token reads all 40 GB, the Framework Desktop manages about 5 tok/s.

Dense models in this brief measure at about 55–70% of the bandwidth ceiling. Speculative decoding (MTP, DFlash, EAGLE and similar) can go above it: a small draft model proposes several tokens and the large model checks them in one pass. AA's configurations ran Qwen3.8-27B 30–120% above the ceiling, and a pinned llama.cpp build took an RTX 3090 from 37.4 to 60.3 tok/s with MTP.530

Prompt processing

Reading the prompt (prefill) is limited by compute instead, because its tokens are processed in parallel. It takes a large share of agent time: AA found prefill took 22–41% of end-to-end time on Qwen3.8-27B, even with 73–93% of prompt tokens served from cache.5 Long context also slows decode. An RTX PRO 6000 generates 210 tok/s on gpt-oss-120b at 4K context and 100 tok/s at 128K.43

Weights · KV cache

Memory math #

Weights need roughly parameters × bits ÷ 8 bytes. A 70B model is about 140 GB at 16-bit and 40–42 GB at Q4_K_M, which averages about 4.8 bits per weight.29 A mixture-of-experts model has to store every parameter, but each token reads only the active ones.

The KV cache comes on top of the weights and grows with context. Its per-token size is 2 (K and V) × layers × KV heads × head dimension × bytes. For Qwen3.8-27B that works out to about 64 KiB per token at f16, so 128K tokens need 8 GiB. One study of quantized reasoning models found a 4-bit KV cache roughly lossless and a 3-bit cache "severe".31

llama.cpp · arXiv 2504.04823 · arXiv 2505.11574 · IJCAI 2025

Quantization #

Quality stays almost flat down to about 4.8 bits per weight, then falls off quickly. At a similar size, Q4_K_M also beats the older Q4_0, because K-quants use a more precise block-scaling scheme.36

Perplexity can miss real damage. "The Illusion of Equivalency" (arXiv 2607.08734) finds that Q8_0 through Q4_K keep accuracy close to the base model yet still produce "noticeable behavioral divergence", and concludes that perplexity "is not a trusted metric".39

Model size and task

Small models and hard tasks lose the most. A comparison of methods at IJCAI 2025 found FP8 "the most reliable method for all model sizes" and AWQ consistently ahead of GPTQ at 70B. It also found quantization "considerably reduces performance in Coding and STEM tasks" on MT-Bench even when leaderboard averages looked stable.38

Tool calling is especially sensitive. Unsloth's write-up, cited by Pinggy, warns that tool calling breaks down below UD-Q2_K_XL. On Qwen3.8-27B, 32-token agreement with the full model fell from about 25% at UD-Q2_K_XL to under 10% at UD-IQ2_S.22

Formats

Prices in USD, late September to mid-October 2026

Hardware and prices #

Unified-memory machines give the most memory per dollar, and NVIDIA's GDDR cards give the most bandwidth per dollar. The 96 GB M5 Ultra has the best bandwidth per dollar of any big-memory machine. At late-2026 street prices the RTX 5090 is no longer cheap VRAM: TensorRigs puts the card alone at about $156 per GB, close to professional cards.72

Every speed on this page is a published measurement with its source named. Sites that estimate speeds can be far off: LLMCheck lists 7 tok/s for gpt-oss-120b on the M5 Max, where MLX measured 88.4973

Two newer options sit near the DGX Spark. NVIDIA's RTX Spark N1X puts a GB10-class chip with 300 GB/s of bandwidth in a Windows laptop; a 128 GB prototype Surface Laptop Ultra ($5,900) finished AA's agent workload in 18.7 minutes.46 AMD's own 128 GB Ryzen AI Halo ($4,000) ran 1.4–1.7× slower than the DGX Spark on three of four models. That is far more than their roughly 7% bandwidth difference, and AA attributes it to the Spark's extra low-precision compute for prefill and its more mature CUDA stack.5 No verified current price was found for DGX Station.

llama.cpp · vLLM · SGLang

Inference engines #

llama.cpp, with Ollama and LM Studio on top of it, is built for one user on whatever hardware is at hand. It splits layers between GPU and CPU so oversized models still load, and it supports the widest range of quantization types.75 vLLM manages the KV cache in pages (PagedAttention) and admits new requests at every step, which is what serving many users at once needs.76

The gap widens with more users. Red Hat measured vLLM producing about 44× more tokens per second than llama.cpp at 64 users on one H200, with llama.cpp's P99 time to first token past 3 minutes.76 With little spare memory the gap closes: on a 24 GB RTX 4090 the two tied at 32 requests (306 vs. 300 tok/s) because vLLM had no VRAM left for batching.79 Two RTX PRO 6000s on vLLM served gpt-oss-120b at 230.5 tok/s to one user and peaked at 664 tok/s across 6 requests.80

Engine by situation

Situation	Engine	Why
One user on a laptop, a Mac or mixed CPU and GPU	llama.cpp, Ollama, LM Studio; MLX on a Mac	Widest hardware and format support, easy setup, CPU offload for oversized models. 81
Fastest single stream on NVIDIA	ExLlamaV2/V3 or TensorRT-LLM	EXL2/EXL3 formats and compiled engines. TensorRT-LLM is the fastest on NVIDIA and takes the most setup. 78
Team or API serving, parallel agents, several GPUs	vLLM or SGLang	Continuous batching, paged KV memory, tensor and expert parallelism. SGLang adds RadixAttention prefix caching and is strong on MoE. 82
Serving on Apple Silicon	vllm-mlx	21–87% higher throughput than llama.cpp and up to 3.7× scaling from continuous batching, per arXiv 2601.19139. 77

HBM · NVLink · batching

Datacenter serving #

Frontier models run on racks of accelerators with high-bandwidth memory. A GB200 or GB300 NVL72 rack links 72 GPUs with 1.8 TB/s of NVLink each (130 TB/s in total) and up to 576 TB/s of aggregate memory bandwidth.8990 Tensor, pipeline and expert parallelism spread one model across many chips.

Batching is the main reason datacenter serving costs less per token. A home machine spends its whole bandwidth budget on one user. A datacenter GPU reads the weights once per step and produces a token for every request in the batch. DeepSeek's published V3/R1 serving stats show each 8×H800 node averaging about 14.8k output tok/s during decoding while each user saw 20–22 tok/s.91

Several other techniques add to this. Running prefill and decode on separate GPU pools gave about 1.5× higher throughput for Llama 3.1 405B in NVIDIA's MLPerf results.92 Prefix caching and KV offload keep repeated context cheap (about 96% of tokens were cached input in AA's Vera Rubin test), and session-aware routing such as NVIDIA Dynamo sends returning users back to their cache. AA measured Vera Rubin NVL72 at 272 million tokens per second per megawatt on DeepSeek V4 Pro, against 239 million for GB300 NVL72, with the lead growing to about 5.7× at around 190 tok/s per user. NVIDIA submitted those results, and the MW figure counts GPU and HBM power only.86

Where Claude runs

Anthropic says it trains and runs Claude on "AWS Trainium, Google TPUs, and NVIDIA GPUs".93

- Project Rainier, about 500,000 Trainium2 chips, came online in October 2025.[94](#s94)
- In October 2025, Anthropic and Google announced access to up to 1 million TPUs, bringing well over a gigawatt online in 2026.[95](#s95)[96](#s96)
- On April 6, 2026, Anthropic announced a Google and Broadcom deal for "multiple gigawatts" of TPU capacity from 2027. Tom's Hardware puts it at about 3.5 GW.[93](#s93)[97](#s97)
- Claude is available on AWS Bedrock, Google Vertex AI and Microsoft Azure Foundry.[93](#s93)

Prices as of October 2026

Recommendations #

For one person, buy enough memory for the model you want, then as much bandwidth as the budget allows. Expect roughly Claude Haiku-class results, about a year behind the frontier. The gap from desk to frontier (about 24 AA points) is twice the gap from open weights to frontier (about 11).

Software defaults

Use Q4_K_M, AWQ or NVFP4 as the default quantization.
Move up to Q5, Q6 or FP8 for coding and math when memory allows.
Avoid anything below about 3.5 bits for agents.
Quantize the KV cache to Q8 for long context.
Turn on speculative decoding.
Switch to vLLM or SGLang only once you serve more than one user.

Caveats #

Anthropic, Qwen and DeepSeek figures are self-reported, use different harnesses and differ from independent runs. Qwen3.8-27B's SWE-bench Pro score of 61.7 uses Qwen's refined version of the benchmark and is not comparable to Anthropic's 89.9%.25 AA Index versions change weights and benchmarks, so only compare scores within v4.3.2. The same hardware can differ 4× by engine and settings, so cross-source speed comparisons are approximate.

Not verified

Parameter counts for MiMo-V2.6-Pro and GLM-5.3
Fable 5.1 and Mythos 5 details beyond the scores shown
Measured speeds for the Mac mini M5 Pro
The price of the 512 GB M5 Ultra
DGX Station pricing
The RTX PRO 6000's $16,000 NVIDIA Marketplace listing, reported by Tom's Hardware, Wccftech and TweakTown but not confirmed by NVIDIA
Reports that NVIDIA ended RTX 5090 production. NVIDIA's Jeff Fisher said on Oct 16 that the card "keeps shipping".98
Qwen3.8-27B's GPQA Diamond score of 89.2, reported by PromptQuorum but not checked against the model card27

Conflicting reports

- M5 Max 128 GB price: $5,099 against $5,800–$6,999.[47](#s47)[48](#s48)
- DGX Spark on AA's Qwen3.8-27B workload: 24.2 min at launch against 17.3 min in the Oct 15 article.[46](#s46)[65](#s65)
- Qwen3.8-Flash-Next memory: "fits 128 GB at 4-bit" (Pinggy) against about 155 GB peak in MacStories' MLX test.[4](#s4)[22](#s22)

getcoretech.net claims Kimi K3 ranked #3 overall, with GLM 5.3 at 59.5 and Opus 5 at 63.1. These do not match AA's published scale and are not charted.28
Bonsai 2 27B, a ternary Qwen3.8-27B, claims parity on HumanEval+ and LiveCodeBench but publishes no agentic benchmarks and needs a llama.cpp fork.25

source & further reading

localai-brief.vercel.app — original article