{"slug": "i-benchmarked-every-model-that-fits-on-an-iphone", "title": "I benchmarked every model that fits on an iPhone", "summary": "An independent benchmark of on-device LLMs on iPhone A17 Pro found Apple's system model achieves ~149 tok/s with only 12MB peak app memory, while 4B-class open models like Qwen3 4B and Llama 3.2 3B trigger thermal throttling, collapsing throughput to 12–24 tok/s. Qwen3 1.7B emerged as the best bring-your-own-model option at ~46 tok/s and 1.2GB peak memory, and 4-bit quantization showed no detectable quality loss on structured tasks.", "body_md": "Every iOS team is having the same argument right now: use Apple's free on-device system model, bring your own open model via MLX, or pay for a cloud API. Most of the arguments run on leaderboard folklore and numbers measured on someone else's hardware. So I measured it myself: the Foundation Models system model plus Qwen3 1.7B/4B/8B, Llama 3.2 3B, and Gemma 3 4B, on the devices they'd actually ship on.\n\n- Apple's system model is the throughput king on iPhone (~149 tok/s) and its weights live\n*outside*your app's memory budget. Peak app memory during inference: 12MB. - There is a thermal cliff at the 4B class on iPhone: Qwen3 4B and Llama 3.2 3B pushed the device to a \"fair\" thermal state mid-run and throughput collapsed to 17–24 tok/s.\n- Qwen3 1.7B is the bring-your-own-model sweet spot on iPhone: 0.33s to first token, ~46 tok/s, 1.2GB peak, and it matched the system model on quality checks.\n- 4-bit quantization cost nothing detectable vs 8-bit on our eval set. The judge-score deltas straddled zero across all five models.\n\n## Method, before the numbers\n\nFixed prompt set, cold and warm runs separated, median of 5, thermal state recorded per\nrun. The harness is a small Swift package (`modelbench`\n\n) that drives\nFoundation Models directly and open models through MLX Swift; quality is checked with a\n5-case golden set: typed tier-1 property assertions plus an LLM judge\n(Qwen3 32B, thinking off, temp 0.1) running locally on a Mac Studio. Judge scores\nare a ranking, not absolutes; the system-model baseline itself moved a point between\nruns, so read ±1–2 as the measurement floor. iPhone numbers are an A17 Pro on OS 26.x;\niPad (M4) numbers are iPadOS 27 beta.\n\n## The iPhone table\n\n| Model | Weights @4-bit | TTFT warm | tok/s (est.) | Peak app mem | Quality |\n|---|---|---|---|---|---|\n| FM system (~3B) | 0 (system) | 0.38s | ~149 | n/a (system) | 5/5 tier-1 · judge 9/10 |\n| Qwen3 1.7B | ~1.0GB | 0.33s | ~46 | 1.2GB | 5/5 tier-1 · judge 10/10 |\n| Llama 3.2 3B | ~1.8GB | 0.73s | ~24 (thermal: fair) | 2.0GB | 5/5 tier-1 · judge 10/10* |\n| Qwen3 4B | ~2.3GB | 1.52s | ~17 (thermal: fair) | 2.4GB | 5/5 tier-1 · judge 8/10 |\n| Gemma 3 4B | ~2.4GB | 1.71s | ~12 | 2.7GB | 5/5 tier-1 · judge 10/10 |\n\n* Llama 3.2 3B judged on 3/5 golden cases (judge-load timeouts on the rest). FM cold TTFT is 2.05s; all open-model runs shown warm. tok/s is estimated from output tokens over generation time, hence \"est.\"\n\n## Finding 1: the system model's real advantage isn't speed, it's memory\n\n~149 tok/s is more than 3× the fastest open model we ran on the same phone, but the\nnumber that changes architectures is the memory one. The system model's weights and KV\ncache are Apple's problem, not yours: our app's peak memory during inference was 12MB.\nEvery MLX model, by contrast, carries its full weight file inside *your* jetsam\nbudget. On an 8GB phone, a 2.4GB model plus your app plus a WebView is a crash report.\n\n## Finding 2: the 4B thermal cliff\n\nThe 4B class technically fits on an A17 Pro. It also cooked the phone. Qwen3 4B and Llama 3.2 3B both degraded the thermal state to \"fair\" during a 5-run median, and sustained throughput landed at 12–24 tok/s, slower reading speed than most users tolerate for streaming UI. The same Qwen3 4B on a fanless iPad (M4) did 36 tok/s at 0.33s TTFT without breaking a sweat. The class boundary is real: on iPhone today, bring a ~2B model or use the system one.\n\n## Finding 3: 4-bit is free (on this task)\n\nChapter-8-of-every-blog wisdom says 4-bit quantization visibly hurts quality. We ran\nthe full golden set at `q8_0`\n\nagainst the Q4 defaults for all five open\nmodels. Tier-1 typed assertions: 5/5 for every model at both quants. Judge-median\ndeltas: −1, −1, +2, 0, 0. That straddles zero with no consistent direction, and ±1–2 is\nthe measurement floor. On a structured-output task, 4-bit cost nothing we could detect,\nand it halves the weight file you're carrying inside your memory budget.\n\n## The Mac numbers, for scale\n\n| Model | Device | TTFT warm | tok/s (est.) | Peak mem |\n|---|---|---|---|---|\n| Qwen3 4B | iPad (M4) | 0.33s | ~36 | 2.5GB |\n| Gemma 3 4B | iPad (M4) | 0.50s | ~37 | 2.8GB |\n| Qwen3 8B | M4 Max | 0.14s | ~105 | 4.8GB |\n| Qwen3 32B | M2 Ultra | 0.84s | ~37 | 17.6GB |\n\nThe 32B row is there because it's the judge for the quality scores above: a local LLM-as-judge on a Mac Studio, no cloud in the eval loop. If your team has one big-RAM Mac on the LAN, you have an eval rig.\n\n## Reproduce it\n\nEvery number here states its method, and the harness builds with\n`swift build`\n\n: no cloud keys, no telemetry. Licenses differ per model\n(Apache-2.0 for Qwen3, Llama Community for 3.2, Gemma Terms for Gemma 3); check them\nbefore shipping weights in an app. If you re-run this on other hardware, I want the\nnumbers: [send them](mailto:ragr1983@gmail.com).\n\nWhere this came from\n\n## This table is Appendix A of a 12-chapter book.\n\n*On-Device Intelligence* covers the decision matrix behind these numbers,\nFoundation Models in production, MLX when you need model control, memory budgets,\nApp Review, and the local eval rig that produced the quality scores. Every code\nsnippet is compiler-verified against the shipping SDK. Chapter 3 is free.\n\n[Get the book](/book/)\n\n[Read Chapter 3 free](/book/ch03-sample.pdf)", "url": "https://wpnews.pro/news/i-benchmarked-every-model-that-fits-on-an-iphone", "canonical_source": "https://digital-foundry-eight.vercel.app/benchmarks/", "published_at": "2026-07-10 01:32:15+00:00", "updated_at": "2026-07-10 02:06:00.209791+00:00", "lang": "en", "topics": ["large-language-models", "ai-products", "ai-infrastructure"], "entities": ["Apple", "Qwen3", "Llama 3.2", "Gemma 3", "MLX", "A17 Pro", "iPad M4"], "alternates": {"html": "https://wpnews.pro/news/i-benchmarked-every-model-that-fits-on-an-iphone", "markdown": "https://wpnews.pro/news/i-benchmarked-every-model-that-fits-on-an-iphone.md", "text": "https://wpnews.pro/news/i-benchmarked-every-model-that-fits-on-an-iphone.txt", "jsonld": "https://wpnews.pro/news/i-benchmarked-every-model-that-fits-on-an-iphone.jsonld"}}