{"slug": "building-local-my-2026-headless-ai-server-journey", "title": "Building Local: My 2026 Headless AI Server Journey", "summary": "A developer reports that running Qwen 3.8 27B at Q5_K_M quantization on a dual AMD Radeon RX 7900 XT and 7800 XT setup achieves 20 tokens per second with a 256k context window, enabling autonomous multi-hour task execution. The 36GB VRAM pool and custom context manager prevent history bloat, and the developer is stress-testing the model to build a fully functional 3D CAD software by the end of the week.", "body_md": "It’s been a while since I last posted my hardware configuration, but local open-weight serving has come a long way. I wanted to share a quick update on how things are running with the **Qwen 3.8 27B** weights at **Q5_K_M quantization**—it’s getting remarkably close to perfect for autonomous, long-horizon work.\n\nMy original single-GPU setup had limitations when pushing heavy context lengths, so I recently added a **second AMD Radeon RX 7900 XT** to pair with my existing **7800 XT**.\n\nThis bumps the total VRAM pool up to a comfortable **36GB**. That extra breathing room is an absolute game-changer, allowing me to comfortably host Qwen 3.8 27B at Q5 while opening up a **256k context window** without running into memory walls.\n\n**Inference Speed:** Sitting right around **20 tokens per second**. While it’s not blindingly fast compared to heavily speculative setups, it is entirely steady and reliable for automated execution.\n\n**Thermals:** Keeping dual AMD cards happy under continuous load takes some tuning. With the right fan profiles and power limits managed via `rocm-smi`\n\n, the 7900 XT memory temperatures stay stably locked **between 80°C and 90°C** even during extended loops.\n\nRunning long instructions over massive context windows natively will eventually cause drift or degradation if left unmanaged. I spent a couple of days engineering a **custom context manager** that sits between the agent and the backend. It keeps the state aggressively clean, preventing the history bloat that usually kills multi-hour sessions.\n\nWith this setup dialed in, the workflow has fundamentally shifted. Instead of interactive back-and-forth chat, I can feed the model a complex, multi-step list of instructions, walk away for a few hours, and come back to a completed task.\n\nRight now, I’m stress-testing the absolute limits of how far this model can go with agentic coding. The current target? Hoping to have a **fully functional 3D CAD software** completely built out through automated instruction loops by the end of the week.\n\nCurious to hear what kind of multi-GPU layer-splitting or context management strategies others in the community are using for the 27B class models on ROCm/llama.cpp right now!", "url": "https://wpnews.pro/news/building-local-my-2026-headless-ai-server-journey", "canonical_source": "https://discuss.huggingface.co/t/building-local-my-2026-headless-ai-server-journey/175243#post_8", "published_at": "2026-08-25 20:23:37+00:00", "updated_at": "2026-08-25 20:44:49.362897+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-agents"], "entities": ["Qwen 3.8 27B", "AMD Radeon RX 7900 XT", "AMD Radeon RX 7800 XT", "ROCm", "llama.cpp"], "alternates": {"html": "https://wpnews.pro/news/building-local-my-2026-headless-ai-server-journey", "markdown": "https://wpnews.pro/news/building-local-my-2026-headless-ai-server-journey.md", "text": "https://wpnews.pro/news/building-local-my-2026-headless-ai-server-journey.txt", "jsonld": "https://wpnews.pro/news/building-local-my-2026-headless-ai-server-journey.jsonld"}}