# Run Qwen3-Coder-Next Locally on a Cost-Effective AI Home PC with llama.cpp > Source: > Published: 2026-09-04 09:03:37+00:00 Running a local language model does not always mean buying a workstation with a huge graphics card. Many cost-effective home PCs already have the parts needed for a useful local AI server: a reasonable CPU, plenty of system memory, an SSD, and a graphics card that can do some of the heavy lifting. In my previous guide, I used Ollama because it makes the first local model easy to run. This time, I want to look at `llama.cpp` and a different kind of model: a Mixture-of-Experts, or MoE, model. The goal is not to claim that an 8 GB graphics card can run every large model at full speed. It cannot. The goal is to show how a cost-effective AI home PC can host a surprisingly capable local model when we make sensible choices about model size, quantization, VRAM, and system RAM. An MoE model can be interesting when you want better answers than a very small model but do not have the budget for a high-end AI GPU. MoE models contain several smaller expert networks. For each part of a prompt, the model routes the work to only some of those experts. This means the model may have a large total parameter count while using a smaller number of active parameters for each token. That can improve the quality-to-speed balance. It does not make the model free to run, though. The model still needs to store its weights somewhere, so system RAM and disk space remain important. Think of a workshop with several specialists. One person is good at code, another is good at writing, and another is good at reasoning. The manager does not ask everyone to work on every small task. It sends each task to the specialists most likely to help. That is the basic idea behind a Mixture-of-Experts model. There are two numbers worth keeping separate: The active number often helps explain why an MoE model can respond efficiently. The total number still matters when planning memory. A quantized model may fit partly in VRAM and partly in system RAM, but it still has to fit somewhere. This guide is aimed at a fairly normal home PC rather than a dedicated server. A comfortable starting point would be: You can try this with 16 GB of RAM, but 32 GB gives the operating system and the model more breathing room. If the model spills into system RAM, response speed will depend heavily on your CPU, memory bandwidth, and storage. For the example in this post, I am using [Qwen3-Coder-Next GGUF](https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF). It is designed for coding agents and has 80 billion total parameters, with about 3 billion active for each token. That is a useful example of why MoE models are interesting for local coding work. There is an important catch. The model page lists the `UD-Q4_K_M` file at roughly 49 GB. An 8 GB GPU and 32 GB of RAM will not be a comfortable Q4 setup. For that machine, start with a smaller 2-bit quantization, or use a smaller MoE model. Treat Qwen3-Coder-Next Q4 as a stretch target for a PC with more than 45 GB of combined RAM and VRAM, plus extra room for the context window. That means an 8 GB GPU will not hold the whole model by itself. That is fine. `llama.cpp` can divide the work between the GPU and CPU. The easiest Windows option is the official package available through WinGet: ``` winget install llama.cpp ``` Close and reopen PowerShell, then check that the command is available: ``` llama --version ``` You can also download a pre-built package from the [llama.cpp releases page](https://github.com/ggml-org/llama.cpp/releases), or build it yourself from the [llama.cpp repository](https://github.com/ggml-org/llama.cpp). The project requires models in the [GGUF format](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). GGUF is simply the model file format used by `llama.cpp` and several other local AI tools. The current `llama.cpp` command can download a compatible model directly from Hugging Face. This is convenient for a first test: ``` llama cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M ``` The first run downloads a large file, so do not be surprised if it takes a while. Make sure you have enough free space before starting. If you prefer to download the file in a browser, save it somewhere simple, such as: ``` D:\models\Qwen3-Coder-Next-UD-IQ2_M.gguf ``` Use the exact filename you downloaded. Quantization names are part of the model choice. A Q4 file normally uses less memory than an 8-bit or 16-bit file, but it may give up a little quality. For a quick terminal test, use: ``` llama cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M -c 4096 ``` For a local service that other applications can use, start `llama serve` : ``` llama serve -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M -c 4096 --host 127.0.0.1 --port 8080 ``` The server provides a browser interface and an OpenAI-compatible API on your own computer. The [llama.cpp server guide](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) explains the available options. Some older Windows packages use the executable name `llama-server.exe` instead. The same idea looks like this when using a local file: ``` llama-server.exe ` -m D:\models\Qwen3-Coder-Next-UD-IQ2_M.gguf ` -c 4096 ` --host 127.0.0.1 ` --port 8080 ``` Keeping the host set to `127.0.0.1` means the service is available only on your computer. That is a sensible default for a home lab. The most important setting is not always the model name. It is how much of the model you ask the GPU to hold. With a recent build of `llama.cpp` , the `-ngl` option controls GPU layer offloading. More layers on the GPU usually means better speed, but it also uses more VRAM. For an 8 GB GPU and 32 GB of RAM, start conservatively: ``` llama serve ` -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M ` -c 4096 ` -ngl 999 ` --cpu-moe ` --host 127.0.0.1 ` --port 8080 ``` The `--cpu-moe` option keeps the MoE expert weights in system RAM. The rest of the model can still use the GPU when there is room. This may be slower than keeping everything in VRAM, but it is a useful way to make a larger model work on a smaller card. For Qwen3-Coder-Next, the model page recommends more than 30 GB of combined RAM and VRAM for its smaller 2-bit XL quantizations. That makes an 8 GB GPU and 32 GB of system RAM a realistic learning setup, although you should expect lower speed than a fully GPU-resident model. If you have 16 GB of VRAM and 32 GB of RAM, the Q4 files become a more realistic experiment, but still leave room for the operating system and context cache. If the process runs out of memory, try these changes one at a time: `4096` to `2048` `-ngl` with a lower number instead of `999` If you have 12 or 16 GB of VRAM, try removing `--cpu-moe` and compare the speed and memory use. Every graphics card, driver, and model build behaves a little differently, so treat these commands as starting points rather than universal benchmarks. Once the server is running, it becomes more than a chat window. You can connect local applications to it and build small generative AI projects around your own data. Here are a few practical home lab ideas: Keep your Docker Compose files, network notes, and maintenance documentation in a private folder. Build a simple question-and-answer tool that helps you find commands and explains how your setup works. Use the local model to summarize manuals, project notes, or long text files without uploading them to a cloud service. Always check the summary against the original document. Point an editor or a small script at the OpenAI-compatible endpoint and use the model to explain code, suggest tests, or help understand an error. It may not replace a cloud coding model, but it is useful for private experiments. Send selected, sanitized log entries to the model and ask it to group repeated errors or explain what to investigate next. Avoid sending passwords, tokens, private keys, or personal information. The next step is retrieval-augmented generation, often called RAG. A small local program can search your notes first, then give the relevant passages to the model as context. This is a good project for learning how search, prompts, and local models fit together. An MoE model on a cost-effective AI home PC can be useful, but it will not feel exactly like a hosted model running on professional hardware. You may see a delay while the model loads. The first response can be slower than later responses. Long conversations use more memory because the model must keep more context available. If the model uses system RAM for expert weights, generation speed may drop noticeably. That is not a failure. A local model is a trade-off between privacy, cost, speed, and quality. For short questions, coding explanations, document summaries, and home lab notes, the trade-off can be very reasonable. Do not compare your tokens-per-second number directly with someone else's unless the model file, quantization, context size, prompt, backend, and hardware are also comparable. Keep the server bound to `127.0.0.1` until you understand authentication and network access. Do not expose it directly to the public internet. Be careful with tools that let a model run shell commands or change files. Start with read-only access and require confirmation before making changes. Also remember that local does not mean automatically correct. A model can invent an answer with great confidence. Verify commands before running them, especially commands involving disks, firewalls, accounts, backups, or deletion. A cost-effective AI home PC can be a useful local AI machine even when its GPU is not designed for large language models. The important part is to plan around the whole computer. VRAM matters, but so do system RAM, CPU speed, storage, cooling, and the model quantization. An MoE model can give you a larger model experience without requiring every parameter to be active for every token, while `llama.cpp` gives you control over how the workload is split between the GPU and CPU. Start with a smaller context, keep the server private, and change one setting at a time. The best setup is not the one with the biggest model. It is the one that runs reliably and is useful enough that you keep using it. In a future post, I will compare a few open MoE models on modest GPUs and look more closely at VRAM usage, CPU offloading, speed, and practical local generative AI workloads.