You’re starting off on the right foot with llama.cpp, as it’s the only tool that uses Vulkan - and Vulkan is the only engine that supports the Vega series. I stuffed quad Vega64s into my ThreadRipper 2950X rig earlier in the year and ran llama.cpp across them (32GB). It wasn’t too optimized, but I quickly found that MoE models are far more efficient on this platform with the limited VRAM.
Qwen2.5 is a very fast set of models - a bit too fast for their own good. It’s only really useful for small things like intellisense and small tasks. I used to run qwen2.5 Coder 1.5B on my CPU+DRAM only, leaving the GPUs free for full models and context.
You should load up Qwen3.8 35B A3B @ Q4. Read up on how Mixture of Experts (MoE) works for this type of model, and the exact llama.cpp parameters to offload MoE. In short, only 3B parameters are active which easily fits within the 8GB VRAM - the rest of the layers will be offloaded to system DRAM. I think you must have offloaded layers in order to get 27B to work anyways, right? The Qwen 27B model is what we call a “Dense” model which is the worse model to use with very limited VRAM - dense models must fit completely into VRAM in order to get speed out of it. However, MoE can be offloaded.
You have a rare motherboard there - it supports native x8 + x8 direct to the CPU. You could pickup another Vega64 for $100 to have 16GB VRAM with layers split.
After playing with Open WebUI, I recommend setting up llama-swap in a docker image. And then configure multiple models to “swap out” when requested. Like Gemma E4B @ Q4 for writing prompts, while switching to Qwen3.8 35B A3B for coding tasks.
Oh, and try to use Docker from now on. You don’t have to compile anything, just run the long commends (by sticking in a bash file).
That 64 GB system ram is what is saving you here to overcome the small 8GB VRAM. Anything you can do to increase that to 16GB, or really 24GB is what you want to target, will blow your mind in what will fit with full context windows. Then it’s a matter of how much you want to pay to increase VRAM even more.
You may also consider upgrading that ram to 2933 or 3200, when you can afford to, as it’s basically 50% faster. Which directly translates to 50% faster tokens, regardless of your setup.
Just don’t spend too much on this setup. No matter how to slice it, you’ll be stuck on Q4 quants and while those work pretty damn well, especially with an agent harness that has a closed loop to analyze its own results and make corrections, will get you pretty far. You can fit Qwen3.8 35B A3B @ Q8 in that DRAM + VRAM for context and any other layers you can fit.
However, if you want more quality and/or less divergence, you really want to plan on larger GPUs with 24-32GB VRAM - two of them - in order to get to the level of running FP8 and FP16 variants for some solid coding tasks. But that’s $2500, minimal, to get started. $1000 for Intel B65 32GB, $1500 AMD R9700 32GB, etc. You can then reserve this Vega64 strictly for image generation using image models as most of those fit. You can also run Qwen2.5 Coder solely on this GPU as well for really fast intellisense, class refactoring, and similar small dedicated tasks. Just remember to turn down Temperature to 0.2 or even 0.0.