It’s been a while since I last posted my hardware configuration, but local open-weight serving has come a long way. I wanted to share a quick update on how things are running with the Qwen 3.8 27B weights at Q5_K_M quantization—it’s getting remarkably close to perfect for autonomous, long-horizon work.
My original single-GPU setup had limitations when pushing heavy context lengths, so I recently added a second AMD Radeon RX 7900 XT to pair with my existing 7800 XT.
This bumps the total VRAM pool up to a comfortable 36GB. That extra breathing room is an absolute game-changer, allowing me to comfortably host Qwen 3.8 27B at Q5 while opening up a 256k context window without running into memory walls.
Inference Speed: Sitting right around 20 tokens per second. While it’s not blindingly fast compared to heavily speculative setups, it is entirely steady and reliable for automated execution.
Thermals: Keeping dual AMD cards happy under continuous load takes some tuning. With the right fan profiles and power limits managed via rocm-smi
, the 7900 XT memory temperatures stay stably locked between 80°C and 90°C even during extended loops.
Running long instructions over massive context windows natively will eventually cause drift or degradation if left unmanaged. I spent a couple of days engineering a custom context manager that sits between the agent and the backend. It keeps the state aggressively clean, preventing the history bloat that usually kills multi-hour sessions.
With this setup dialed in, the workflow has fundamentally shifted. Instead of interactive back-and-forth chat, I can feed the model a complex, multi-step list of instructions, walk away for a few hours, and come back to a completed task.
Right now, I’m stress-testing the absolute limits of how far this model can go with agentic coding. The current target? Hoping to have a fully functional 3D CAD software completely built out through automated instruction loops by the end of the week.
Curious to hear what kind of multi-GPU layer-splitting or context management strategies others in the community are using for the 27B class models on ROCm/llama.cpp right now!