{"slug": "qwen3-8-27b-on-the-iphone-18-neural-engine-at-15-70-tok-s-1-5-bpw", "title": "Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)", "summary": "Anemll reported running the Qwen 3.8-27B model on the iPhone 18's Apple A20 Neural Engine at 15–60 tokens per second, peaking at 70 tok/s, using a base model requantized from PrismML's Bonsai-2 at 1.585 bpw down to 1.5 bpw (6-bit vector quantization per 4 weights). Anemll confirmed a 24K context window on the iPhone with both K/V caches in int8 while the delta net stays FP16, and added a dflash-2 draft model at roughly 2.8 bpw with mixed 2–4 vector quantization and a small drop in acceptance rate. Anemll wrote that \"Apple Neural Engine on A20 is fast.", "body_md": "Anemll on X: \"Confirmed 24K context on iPhone with both K/V on int8. Delta net stays FP16. Decoding also peaks at 70 tok/s with easy to predict context.\nApple Neural Engine on A20 is fast 🚀\"\n\nConfirmed 24K context on iPhone with both K/V on int8. Delta net stays FP16. Decoding also peaks at 70 tok/s with easy to predict context.\nApple Neural Engine on A20 is fast 🚀\n\nQwen 3.8- 27B on iPhone 18 Apple Neural Engine.\nAdded draft model , dflash-2 at ~ 2.8 bpw ( mixed 2-4 vector quantization with small drop in acceptance rate)\nBase is @PrismML Bonsai-2, 1.585 bpw requantized to 1.5 bpw ( VQ 6 bits per 4 weights)\nDecoding speed is 15-60 tps.\n\nConfirmed 24K context on iPhone with both K/V on int8. Delta net stays FP16. Decoding also peaks at 70 tok/s with easy to predict context.\nApple Neural Engine on A20 is fast 🚀\n\nQwen 3.8- 27B on iPhone 18 Apple Neural Engine.\nAdded draft model , dflash-2 at ~ 2.8 bpw ( mixed 2-4 vector quantization with small drop in acceptance rate)\nBase is @PrismML Bonsai-2, 1.585 bpw requantized to 1.5 bpw ( VQ 6 bits per 4 weights)\nDecoding speed is 15-60 tps.", "url": "https://wpnews.pro/news/qwen3-8-27b-on-the-iphone-18-neural-engine-at-15-70-tok-s-1-5-bpw", "canonical_source": "https://twitter.com/anemll/status/2111517143087067309", "published_at": "2026-10-17 18:10:21+00:00", "updated_at": "2026-10-17 18:37:54.945852+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "machine-learning", "ai-research"], "entities": ["Anemll", "Qwen 3.8-27B", "iPhone 18", "Apple A20 Neural Engine", "PrismML", "Bonsai-2", "dflash-2"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/qwen3-8-27b-on-the-iphone-18-neural-engine-at-15-70-tok-s-1-5-bpw", "markdown": "https://wpnews.pro/news/qwen3-8-27b-on-the-iphone-18-neural-engine-at-15-70-tok-s-1-5-bpw.md", "text": "https://wpnews.pro/news/qwen3-8-27b-on-the-iphone-18-neural-engine-at-15-70-tok-s-1-5-bpw.txt", "jsonld": "https://wpnews.pro/news/qwen3-8-27b-on-the-iphone-18-neural-engine-at-15-70-tok-s-1-5-bpw.jsonld"}}