Earlier, I wrote about Qwen 3.8 being good but slow. Since then, I have been trying to figure out how to boost the speed of running Qwen 3.8 on my 64GB M2 Ultra Mac Studio.
What is working for me is mlx-serve. This is kinda Ollama-ish, in that it has an inference server with an associated GUI. Overall, it does quite a bit more than Ollama does. But the biggest thing is that it runs Qwen 3.8
On the one hand, that means that Sonnet is much faster and gives higher-quality results. On the other hand, Qwen 3.8 is a local model:
Right now, I am having Sonnet review Qwen's work, but so far, Qwen has reliably delivered what it was supposed to do. And this is about a 4x speed increase from what I had with my Ollama-based Qwen 3.8 setup.
Part of that speed increase may be from better MLX support in mlx-serve
than in Ollama. The bigger thing might be the support that mlx-serve
has for MTP (Multi-Token Prediction). MTP is a form of "speculative decoding", where lesser models do the heavy lifting and the top-tier model chooses among their outputs. Qwen 3.6 and 3.8 offer MTP, and mlx-serve
lets me toggle that on.
There are clear limits. Having Qwen 3.8 do agentic coding consumes 99% of the GPU of my Mac Studio, and that is with just a single conversation. Agent swarms require a fleet of hardware, not just a single Mac.
Still, this is more than good enough for lightweight tasks and lets me offload some of my development from frontier models and their data centers.