Local AI means running AI models directly on hardware you own, like a laptop, workstation or on-prem server, instead of calling a cloud API.
This report is a monthly updated overview of what is possible with local AI, with which models, and on what hardware. What you can run locally is decided by memory capacity (does the model fit?), memory bandwidth (how fast do tokens come out?) and compute (how fast are long prompts ingested?). This report maps capabilities to model classes, model classes to hardware, and hardware to realistic tokens per second.
Released in August: dense, Apache 2.0, native vision-language input and 262K context, about 17 GB at Q4.
A 30B model tuned for agentic tool-calling loops on your own hardware, with quantized builds under 20 GB.
Launched August 14 on the API; the flagship weights landed on Hugging Face August 28 under a bespoke GLM-5.3 license. GLM-5.3-Flash shipped August 26 under MIT: a 320B-total / 18B-active MoE whose 3-bit quant (~120 GB) fits the 128 GB unified-memory tier.
Announced August 25, shipping September 22 (the 512 GB config late October). The M5 Ultra lifts memory bandwidth from 819 GB/s to 1.2 TB/s, and Thunderbolt 5 RDMA now clusters multiple Studios: Apple claims four systems deliver up to 3x the inference of one.
Prefill processes your prompt (all input tokens at once, in parallel) and ends when the first output token appears. Decode then generates the answer one token at a time. The two phases stress different parts of the hardware, so a machine can be fast at one and slow at the other.
Bigger models need more memory, plus headroom that grows with longer context. A mid-size model takes about 16 GB; a large one needs 40 GB or more and stops fitting ordinary hardware.
Generating each token requires reading all active weights from memory once, and there is no way to parallelize across a single stream. The GPU's arithmetic units sit idle waiting on memory, so tokens/sec tracks GB/s almost linearly: an RTX 5090 (1,792 GB/s) does ~142 tok/s on an 8B, a DGX Spark (273 GB/s) a fraction of that on the same model.
The whole prompt is processed in parallel, so weights are read once and reused across many tokens. Here raw compute is the limit: a discrete GPU like the RTX 5090 does 10,000+ tok/s prefill on an 8B, an order of magnitude ahead of compact unified-memory machines. For RAG and long-context work this gap often dominates wall-clock time.
Mixture-of-Experts models store all experts but activate only a few billion parameters per token. gpt-oss-120b is 116.8B total / 5.1B active, ships at 60.8 GB in MXFP4, and runs 30 to 55 tok/s on a 128 GB unified-memory desktop (Strix Halo, DGX Spark). A dense 70B on the same machines crawls at 3 to 5 tok/s.
Chat and writing are decode-heavy, so memory bandwidth determines the experience (Macs and unified-memory boxes do well). RAG, long documents and agentic coding are prefill-heavy, so compute determines it (discrete GPUs are far ahead). Capacity is a hard gate for both: if the model doesn't fit, the rest is moot.
| Use case | Threshold | Why |
|---|---|---|
| Reading-speed chat | 10-30 tok/s | Human reading is ~7 to 15 tok/s; 15+ feels comfortable |
| Coding assistant | 30+ tok/s | Plus fast prefill; large-context prompts need quick time-to-first-token |
| Agentic loops | 50+ tok/s | Multi-step tool calls multiply token counts; reasoning models emit thousands of thinking tokens |
| Batch / overnight | <10 tok/s ok | Throughput per dollar matters more than latency |
| Capability | Minimum | Recommended | Notes |
|---|---|---|---|
| Basic chat / Q&A | 1-4B Gemma 4 E4BLlama 3.2 3BQwen3 4B | 7-9B Llama 3.1 8BQwen3 8B | Small models handle simple factual chat but hallucinate more |
| Summarization | 3-4B Gemma 4 E4BPhi-4-mini | 8-14B Llama 3.1 8BQwen3 14B | Long documents need context window plus prefill speed |
| Writing assistance | 7-8B Mistral 7BLlama 3.1 8B | 12-31B Gemma 4 31BQwen3.8-27B | The 2026 27-31B dense class is very strong here |
| Translation | 7-8B multilingual Qwen3 8BMistral 7B | 27-31B Gemma 4 31BQwen3.8-27B | Gemma 4 covers 140+ languages, Qwen 100+ |
| Code autocomplete | 3-7B Qwen2.5-Coder 7B | 7-14B Qwen2.5-Coder 14B | Latency matters more than size; target sub-500 ms |
| Agentic coding | 27-32B Qwen3.8-27BQwen3-Coder-Next | 27B+ dense or MoE Qwen3.8-27Bgpt-oss-120bGLM-5.3 | Qwen3.6-27B hit 77.2% SWE-bench Verified from a dense 27B, and Qwen3.8-27B improves on it, making one 24-32 GB GPU viable for daily agentic coding. GLM-5.3 (weights released late August under a bespoke license) is the open-weight server-class leader |
| RAG over documents | 7-8B + embeddings Llama 3.1 8BNomic Embed | 12-32B Qwen3.8-27BGemma 4 26B A4B | Prefill speed and context window are the real constraints |
| Long-context analysis | 8B · 128K Llama 3.1 8BQwen3 8B | 27-31B · 256K+ Qwen3.8-27BGemma 4 31B | Qwen3.8-27B has 262K native context; Gemma 4 12B+ models have 256K (E2B/E4B: 128K). KV cache grows with context and needs memory headroom |
| Tool use / agents | 8B trained for it Qwen3 8BLlama 3.1 8B | 27-31B+ Qwen3.8-27BGemma 4 31Bgpt-oss-120b | Reliability jumps near 30B; Qwen3.8, Gemma 4, gpt-oss are tool-trained |
| Vision (image understanding) | 2-4B VLM Gemma 4 E4B | 26-31B VLM Qwen3.8-27BGemma 4 31B | All Gemma 4 sizes are multimodal; Qwen3.8-27B ships a vision encoder |
| Image generation | ~1B diffusion · 4 GB SD 1.5 | 12B diffusion · 12-24 GB FLUX.1 Dev | Diffusion models, not LLMs; sized by VRAM. FLUX.2 Dev (32B) needs 24 GB+ with quantization/offload, best on 48-96 GB |
| Speech-to-text (live) | 244M · CPU ok Whisper small | 1.5B · ~3 GB INT8 Whisper large-v3 | Very VRAM-friendly; real-time factor is the metric |
| Audio transcription (recorded) | 0.2-0.6B · CPU Whisper smallParakeet TDT 0.6B | 809M Whisper large-v3-turbo | Batch work, so interactive speed thresholds don't apply; recordings process faster than real time even on modest hardware |
| Text-to-speech | 82M · CPU Kokoro-82M | 4-12 GB XTTS-v2Fish Speech | XTTS is non-commercial; Kokoro (Apache 2.0) is the permissive default |
| Fine-tuning (QLoRA) | 7-8B on 12-16 GB Mistral 7BLlama 3.1 8B | to ~14B on 24 GB Qwen3 14BGemma 3 12B | QLoRA: 7B ~12 GB, 13B ~20 GB, 70B ~88 GB. Full fine-tuning stays in the datacenter |
Q4 ≈ 0.6 GB and Q8 ≈ 1.1 GB per billion parameters, plus KV cache. gpt-oss-120b, DeepSeek V4-Flash and Kimi K3 ship natively in low precision (MXFP4 / FP4+FP8); no Q8 build applies.
| Profile | Setup | Cost | Upgrade trigger |
|---|---|---|---|
| Laptop: chat, autocomplete, light RAG | 16-32 GB laptop (MacBook Air M5 or a 16 GB+ Windows machine). Runs 4-8B models at reading speed for chat, autocomplete and summarization; a 32 GB config also fits 7-14B at reading speed. Larger models fit but fall below usable speed, and nothing in this tier reaches coding-assistant pace. | $1,300-2,200 | You want 14B+ models at usable speed, or coding-assistant prefill and decode. |
| Solo developer: fast agentic coding | 24-32 GB GPU (RTX 4090/5090) running Qwen3.8-27B or Gemma 4 31B at Q4. Fastest prefill per dollar. | $3,000-6,500 | You need models beyond 32B, which is a capacity problem, not a speed one. |
| Big-model capacity: 128 GB unified memory | AMD Strix Halo box, NVIDIA DGX Spark, or 128 GB MacBook Pro M5 Max. Runs MoE like gpt-oss-120b at 31-55 tok/s, or GLM-5.3-Flash at 3-bit; two DGX Sparks pair over 200 GbE into 256 GB for models neither fits alone. | $4,000-9,400 | You need higher single-stream speed or frontier-class models. |
| Small team: frontier open models | Mac Studio M5 Ultra 256-512 GB (DeepSeek V4-Flash native, quantized GLM-5.2/5.3 on 512 GB) for quiet low power, or RTX Pro 6000 96 GB for prefill speed and concurrency. M5 Ultra ships Sept 22; the 512 GB config lands late October. | $9,500-40,000 | You need concurrency for many users. |
| Production: many users, trillion-parameter | 8x H100/H200 node. GLM-5.3 and DeepSeek V4-Pro are the open-weight targets here. | $150,000+ | Sustained multi-user volume that the workstation tiers can't serve. |