Every iOS team is having the same argument right now: use Apple's free on-device system model, bring your own open model via MLX, or pay for a cloud API. Most of the arguments run on leaderboard folklore and numbers measured on someone else's hardware. So I measured it myself: the Foundation Models system model plus Qwen3 1.7B/4B/8B, Llama 3.2 3B, and Gemma 3 4B, on the devices they'd actually ship on.
- Apple's system model is the throughput king on iPhone (~149 tok/s) and its weights live outsideyour app's memory budget. Peak app memory during inference: 12MB. - There is a thermal cliff at the 4B class on iPhone: Qwen3 4B and Llama 3.2 3B pushed the device to a "fair" thermal state mid-run and throughput collapsed to 17–24 tok/s.
- Qwen3 1.7B is the bring-your-own-model sweet spot on iPhone: 0.33s to first token, ~46 tok/s, 1.2GB peak, and it matched the system model on quality checks.
- 4-bit quantization cost nothing detectable vs 8-bit on our eval set. The judge-score deltas straddled zero across all five models.
Method, before the numbers #
Fixed prompt set, cold and warm runs separated, median of 5, thermal state recorded per
run. The harness is a small Swift package (modelbench
) that drives Foundation Models directly and open models through MLX Swift; quality is checked with a 5-case golden set: typed tier-1 property assertions plus an LLM judge (Qwen3 32B, thinking off, temp 0.1) running locally on a Mac Studio. Judge scores are a ranking, not absolutes; the system-model baseline itself moved a point between runs, so read ±1–2 as the measurement floor. iPhone numbers are an A17 Pro on OS 26.x;
iPad (M4) numbers are iPadOS 27 beta.
The iPhone table #
| Model | Weights @4-bit | TTFT warm | tok/s (est.) | Peak app mem | Quality |
|---|---|---|---|---|---|
| FM system (~3B) | 0 (system) | 0.38s | ~149 | n/a (system) | 5/5 tier-1 · judge 9/10 |
| Qwen3 1.7B | ~1.0GB | 0.33s | ~46 | 1.2GB | 5/5 tier-1 · judge 10/10 | | Llama 3.2 3B | ~1.8GB | 0.73s | ~24 (thermal: fair) | 2.0GB | 5/5 tier-1 · judge 10/10* | | Qwen3 4B | ~2.3GB | 1.52s | ~17 (thermal: fair) | 2.4GB | 5/5 tier-1 · judge 8/10 | | Gemma 3 4B | ~2.4GB | 1.71s | ~12 | 2.7GB | 5/5 tier-1 · judge 10/10 |
- Llama 3.2 3B judged on 3/5 golden cases (judge-load timeouts on the rest). FM cold TTFT is 2.05s; all open-model runs shown warm. tok/s is estimated from output tokens over generation time, hence "est."
Finding 1: the system model's real advantage isn't speed, it's memory #
~149 tok/s is more than 3× the fastest open model we ran on the same phone, but the number that changes architectures is the memory one. The system model's weights and KV cache are Apple's problem, not yours: our app's peak memory during inference was 12MB. Every MLX model, by contrast, carries its full weight file inside your jetsam budget. On an 8GB phone, a 2.4GB model plus your app plus a WebView is a crash report.
Finding 2: the 4B thermal cliff #
The 4B class technically fits on an A17 Pro. It also cooked the phone. Qwen3 4B and Llama 3.2 3B both degraded the thermal state to "fair" during a 5-run median, and sustained throughput landed at 12–24 tok/s, slower reading speed than most users tolerate for streaming UI. The same Qwen3 4B on a fanless iPad (M4) did 36 tok/s at 0.33s TTFT without breaking a sweat. The class boundary is real: on iPhone today, bring a ~2B model or use the system one.
## Finding 3: 4-bit is free (on this task)
Chapter-8-of-every-blog wisdom says 4-bit quantization visibly hurts quality. We ran
the full golden set at q8_0
against the Q4 defaults for all five open models. Tier-1 typed assertions: 5/5 for every model at both quants. Judge-median deltas: −1, −1, +2, 0, 0. That straddles zero with no consistent direction, and ±1–2 is the measurement floor. On a structured-output task, 4-bit cost nothing we could detect, and it halves the weight file you're carrying inside your memory budget.
The Mac numbers, for scale #
| Model | Device | TTFT warm | tok/s (est.) | Peak mem |
|---|---|---|---|---|
| Qwen3 4B | iPad (M4) | 0.33s | ~36 | 2.5GB |
| Gemma 3 4B | iPad (M4) | 0.50s | ~37 | 2.8GB |
| Qwen3 8B | M4 Max | 0.14s | ~105 | 4.8GB |
| Qwen3 32B | M2 Ultra | 0.84s | ~37 | 17.6GB |
The 32B row is there because it's the judge for the quality scores above: a local LLM-as-judge on a Mac Studio, no cloud in the eval loop. If your team has one big-RAM Mac on the LAN, you have an eval rig.
Reproduce it #
Every number here states its method, and the harness builds with
swift build
: no cloud keys, no telemetry. Licenses differ per model (Apache-2.0 for Qwen3, Llama Community for 3.2, Gemma Terms for Gemma 3); check them before shipping weights in an app. If you re-run this on other hardware, I want the
numbers: send them. Where this came from
This table is Appendix A of a 12-chapter book. #
On-Device Intelligence covers the decision matrix behind these numbers, Foundation Models in production, MLX when you need model control, memory budgets, App Review, and the local eval rig that produced the quality scores. Every code snippet is compiler-verified against the shipping SDK. Chapter 3 is free.
[Get the book](/book/)
[Read Chapter 3 free](/book/ch03-sample.pdf)