I really need to just put this in a reference post somewhere so I can just point to it…
Your answer is, “it all depends”. What do you want to do? For me, I have a Strix Halo GMKtec box with 128 gig of unified memory. I run Qwen/Qwen3-Coder-Next-GGUF:Q8_0 on it at over 44 tps. The context window is maxed out at 262144. But i have nice long chats with Claude code, so I need the largest context window that I can get. Bless Wendell, he sold me on the box when it as $1999. And I know that I’m slow, but 44 tps doesn’t make me wait too long. Also, I have had cloud based interactions that don’t come close to the Strix Halo.
So I also have a crappy old 3060 NVIDIA card with 12 gig of memory. So I can run basic stuff on that as well. My e-mailer AI filter agent runs just fine. In vscode, I have the autocomplete to also point to the small, local instance of qwen35. I was doing a README.md file the other day, the autocomplete coming off of a 8gig qwen35 model was pretty impressive.
Finally, I do love the tool use. Running Claude locally. I use Cline and Continue in VScode. Just doing the git work…. As asked Claude the other night to generate an antenna pattern, call gnuplot, and spit out a PNG file. It wasn’t 20 seconds. I had never asked it to use gnuplot or produce a PNG file before.
So, back in the dark ages, about two months ago, a 27 gig model that is designed to be loaded into 32 gig of memory, You could do a lot of the local stuff. SIMPLE tools use wasn’t that bad. Don’t expect much, but it can certainly create directories and do builds and interact with github.
From what I have seen, qwen 3.8 27B has made major improvements in tool use. A 32 gig card would be much faster than the Strix Halo (it has a 110 Watt power limit, can’t ask too much), and probably do much of what someone could want. My friend was over the other day with his Mac with 24 gig of unified memory. We put ollama on it and a 9 gig qwen3.9 27B quant. It was actually very fast. He was working away in VScode. Made his fans come on for the first time. But it was nice. Great simple tool use, very good autocomplete. They have come very far. But, you are not going to have very long planning chats. He did the generate a TIC-TAC-TOE demo and it just barely made it in context window.
I also have an AMD account and an Anthropic account. The other night, I asked the agent to add OpenMP to my simulator project. That did not go well. After about 15 minutes, it blew out the context and fell over in a heap. It was fun watching it desperately making changes and running build and test cycles. So then I can ask Anthropic. Don’t need to send my emails there, don’t need my autocomplete going there, the local models can check in code and do pull requests. Even have decent planning sessions.
So what do you want? Serve an office of 50 developers? Just local services? You can get a whole lot done on 32 gig. When I got the Strix Halo at $1999, I was thinking…. well, we’ll see. Now, I’m thrilled with it, but it is now $3600.
How important is speed to you? How many models do you want to run at the same time - it isn’t one. I only run code assist models. My friend with the Mac also runs image generation models (that I know nothing about). It all has to fit into memory.
My 2 cents