While we were trying to solve a separate problem, we ended up on a bit of a side quest and built an inference engine. We got ourselves some nice, shiny new Mac M5 Max 128GB machines while doing a lot of local LLM work for another project and ended up wanting to run some bigger models.
When we were running models for coding, we noticed a few improvements that we might be able to make around repetition between turns. After that, the whole thing got a little out of hand, and we ended up writing our own Metal engine with native multi-token prediction.
The runtime is open source under Apache 2.0. I'd love feedback on how we've put the engine together, especially from people working with Metal or local LLMs who have an M5 Max.
We've only tested this locally, so it would be great if anyone could give it a prod and help us work out whether we are going crazy or not.
Comments URL: [https://news.ycombinator.com/item?id=50438931](https://news.ycombinator.com/item?id=50438931)
Points: 1