Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)

Anemll reported running the Qwen 3.8-27B model on the iPhone 18's Apple A20 Neural Engine at 15–60 tokens per second, peaking at 70 tok/s, using a base model requantized from PrismML's Bonsai-2 at 1.585 bpw down to 1.5 bpw (6-bit vector quantization per 4 weights). Anemll confirmed a 24K context window on the iPhone with both K/V caches in int8 while the delta net stays FP16, and added a dflash-2 draft model at roughly 2.8 bpw with mixed 2–4 vector quantization and a small drop in acceptance rate. Anemll wrote that "Apple Neural Engine on A20 is fast.

Anemll on X: "Confirmed 24K context on iPhone with both K/V on int8. Delta net stays FP16. Decoding also peaks at 70 tok/s with easy to predict context. Apple Neural Engine on A20 is fast 🚀" Confirmed 24K context on iPhone with both K/V on int8. Delta net stays FP16. Decoding also peaks at 70 tok/s with easy to predict context. Apple Neural Engine on A20 is fast 🚀 Qwen 3.8- 27B on iPhone 18 Apple Neural Engine. Added draft model , dflash-2 at ~ 2.8 bpw mixed 2-4 vector quantization with small drop in acceptance rate Base is @PrismML Bonsai-2, 1.585 bpw requantized to 1.5 bpw VQ 6 bits per 4 weights Decoding speed is 15-60 tps. Confirmed 24K context on iPhone with both K/V on int8. Delta net stays FP16. Decoding also peaks at 70 tok/s with easy to predict context. Apple Neural Engine on A20 is fast 🚀 Qwen 3.8- 27B on iPhone 18 Apple Neural Engine. Added draft model , dflash-2 at ~ 2.8 bpw mixed 2-4 vector quantization with small drop in acceptance rate Base is @PrismML Bonsai-2, 1.585 bpw requantized to 1.5 bpw VQ 6 bits per 4 weights Decoding speed is 15-60 tps.