I benchmarked every model that fits on an iPhone

An independent benchmark of on-device LLMs on iPhone A17 Pro found Apple's system model achieves ~149 tok/s with only 12MB peak app memory, while 4B-class open models like Qwen3 4B and Llama 3.2 3B trigger thermal throttling, collapsing throughput to 12–24 tok/s. Qwen3 1.7B emerged as the best bring-your-own-model option at ~46 tok/s and 1.2GB peak memory, and 4-bit quantization showed no detectable quality loss on structured tasks.

Every iOS team is having the same argument right now: use Apple's free on-device system model, bring your own open model via MLX, or pay for a cloud API. Most of the arguments run on leaderboard folklore and numbers measured on someone else's hardware. So I measured it myself: the Foundation Models system model plus Qwen3 1.7B/4B/8B, Llama 3.2 3B, and Gemma 3 4B, on the devices they'd actually ship on. - Apple's system model is the throughput king on iPhone ~149 tok/s and its weights live outside your app's memory budget. Peak app memory during inference: 12MB. - There is a thermal cliff at the 4B class on iPhone: Qwen3 4B and Llama 3.2 3B pushed the device to a "fair" thermal state mid-run and throughput collapsed to 17–24 tok/s. - Qwen3 1.7B is the bring-your-own-model sweet spot on iPhone: 0.33s to first token, ~46 tok/s, 1.2GB peak, and it matched the system model on quality checks. - 4-bit quantization cost nothing detectable vs 8-bit on our eval set. The judge-score deltas straddled zero across all five models. Method, before the numbers Fixed prompt set, cold and warm runs separated, median of 5, thermal state recorded per run. The harness is a small Swift package modelbench that drives Foundation Models directly and open models through MLX Swift; quality is checked with a 5-case golden set: typed tier-1 property assertions plus an LLM judge Qwen3 32B, thinking off, temp 0.1 running locally on a Mac Studio. Judge scores are a ranking, not absolutes; the system-model baseline itself moved a point between runs, so read ±1–2 as the measurement floor. iPhone numbers are an A17 Pro on OS 26.x; iPad M4 numbers are iPadOS 27 beta. The iPhone table | Model | Weights @4-bit | TTFT warm | tok/s est. | Peak app mem | Quality | |---|---|---|---|---|---| | FM system ~3B | 0 system | 0.38s | ~149 | n/a system | 5/5 tier-1 · judge 9/10 | | Qwen3 1.7B | ~1.0GB | 0.33s | ~46 | 1.2GB | 5/5 tier-1 · judge 10/10 | | Llama 3.2 3B | ~1.8GB | 0.73s | ~24 thermal: fair | 2.0GB | 5/5 tier-1 · judge 10/10 | | Qwen3 4B | ~2.3GB | 1.52s | ~17 thermal: fair | 2.4GB | 5/5 tier-1 · judge 8/10 | | Gemma 3 4B | ~2.4GB | 1.71s | ~12 | 2.7GB | 5/5 tier-1 · judge 10/10 | Llama 3.2 3B judged on 3/5 golden cases judge-load timeouts on the rest . FM cold TTFT is 2.05s; all open-model runs shown warm. tok/s is estimated from output tokens over generation time, hence "est." Finding 1: the system model's real advantage isn't speed, it's memory ~149 tok/s is more than 3× the fastest open model we ran on the same phone, but the number that changes architectures is the memory one. The system model's weights and KV cache are Apple's problem, not yours: our app's peak memory during inference was 12MB. Every MLX model, by contrast, carries its full weight file inside your jetsam budget. On an 8GB phone, a 2.4GB model plus your app plus a WebView is a crash report. Finding 2: the 4B thermal cliff The 4B class technically fits on an A17 Pro. It also cooked the phone. Qwen3 4B and Llama 3.2 3B both degraded the thermal state to "fair" during a 5-run median, and sustained throughput landed at 12–24 tok/s, slower reading speed than most users tolerate for streaming UI. The same Qwen3 4B on a fanless iPad M4 did 36 tok/s at 0.33s TTFT without breaking a sweat. The class boundary is real: on iPhone today, bring a ~2B model or use the system one. Finding 3: 4-bit is free on this task Chapter-8-of-every-blog wisdom says 4-bit quantization visibly hurts quality. We ran the full golden set at q8 0 against the Q4 defaults for all five open models. Tier-1 typed assertions: 5/5 for every model at both quants. Judge-median deltas: −1, −1, +2, 0, 0. That straddles zero with no consistent direction, and ±1–2 is the measurement floor. On a structured-output task, 4-bit cost nothing we could detect, and it halves the weight file you're carrying inside your memory budget. The Mac numbers, for scale | Model | Device | TTFT warm | tok/s est. | Peak mem | |---|---|---|---|---| | Qwen3 4B | iPad M4 | 0.33s | ~36 | 2.5GB | | Gemma 3 4B | iPad M4 | 0.50s | ~37 | 2.8GB | | Qwen3 8B | M4 Max | 0.14s | ~105 | 4.8GB | | Qwen3 32B | M2 Ultra | 0.84s | ~37 | 17.6GB | The 32B row is there because it's the judge for the quality scores above: a local LLM-as-judge on a Mac Studio, no cloud in the eval loop. If your team has one big-RAM Mac on the LAN, you have an eval rig. Reproduce it Every number here states its method, and the harness builds with swift build : no cloud keys, no telemetry. Licenses differ per model Apache-2.0 for Qwen3, Llama Community for 3.2, Gemma Terms for Gemma 3 ; check them before shipping weights in an app. If you re-run this on other hardware, I want the numbers: send them mailto:ragr1983@gmail.com . Where this came from This table is Appendix A of a 12-chapter book. On-Device Intelligence covers the decision matrix behind these numbers, Foundation Models in production, MLX when you need model control, memory budgets, App Review, and the local eval rig that produced the quality scores. Every code snippet is compiler-verified against the shipping SDK. Chapter 3 is free. Get the book /book/ Read Chapter 3 free /book/ch03-sample.pdf