After checking out Turbo Fieldfare (a very interesting project that streamed experts from SSD on Apple Silicon Macs to run Gemma 4 26B with very little RAM usage), I tried to make this project much faster and succeeded!
This project gets up to 100+ tokens/s inference on Gemma 4 26B (Turbo Fieldfare runs at 20-30 on similar hardware) and doesn't have insanely slow prefill times due to the ANE helping (I had to reverse engineer Apple's APIs to get this working).
I also implemented speculative decoding, and while Turbo steered away from using mmap, this repo utilizes it so the OS file cache serves as the expert cache.
Comments URL: [https://news.ycombinator.com/item?id=50174998](https://news.ycombinator.com/item?id=50174998)
Points: 1
source & further reading
github.com — original article
A script that tests whether your network breaks large uploads to the Claude API
Show HN: Melra – pause ambiguous MCP writes instead of blindly retrying
Foundry-protocol – A framework for orchestrating complex tasks with AI agents