Show HN: Gemma 4 26B at 100 tok/s on a Mac, using the Neural Engine and 2GB RAM

A developer reports running Gemma 4 26B at up to 100+ tokens per second on a Mac with 2GB of RAM, using Apple's Neural Engine (ANE) and reverse-engineered Apple APIs, compared with 20-30 tokens per second for the Turbo Fieldfare project on similar hardware. The project also implements speculative decoding and uses mmap so the OS file cache serves as the expert cache, avoiding the slow prefill times the developer attributes to the ANE-assisted approach. The work follows Turbo Fieldfare's method of streaming experts from SSD on Apple Silicon Macs.

After checking out Turbo Fieldfare a very interesting project that streamed experts from SSD on Apple Silicon Macs to run Gemma 4 26B with very little RAM usage , I tried to make this project much faster and succeeded This project gets up to 100+ tokens/s inference on Gemma 4 26B Turbo Fieldfare runs at 20-30 on similar hardware and doesn't have insanely slow prefill times due to the ANE helping I had to reverse engineer Apple's APIs to get this working . I also implemented speculative decoding, and while Turbo steered away from using mmap, this repo utilizes it so the OS file cache serves as the expert cache. Comments URL: https://news.ycombinator.com/item?id=50174998 https://news.ycombinator.com/item?id=50174998 Points: 1 Comments: 0