Running Qwen3.8–27B on an RTX 4090

A hands-on test by Kwangju Shin (KJ) ran Qwen3.8-27B on a single RTX 4090 at roughly 100 tokens per second on short, cached-input prompts, with a complete prose response arriving in an average of 1.68 seconds. The setup paired the Unsloth Qwen3.8-27B-UD-Q3_K_XL.gguf main model with Inco AI's Qwen3.8-27B-DFlash2-Q4_K_M.gguf draft model for speculative decoding, a quantized KV cache, and llama.cpp commit 1deefcc with --spec-type draft-dflash and --spec-draft-n-max 4. Shin noted the final speed belongs to the whole configuration and that no matched drafting-on versus drafting-off comparison was run.

Kwangju Shin KJ Could I run Qwen3.8–27B at around 100 tokens per second on a single RTX 4090? In my short, cached-input tests, I could. A complete prose response arrived in an average of 1.68 seconds. I wanted a local LLM that an agent could call from my own PC. Speed was the first question: before asking it to carry out a sequence of tasks, I wanted to know how quickly this model could respond on the GPU I had. The configuration paired Qwen3.8–27B with a DFlash2 draft model, using smaller weights and a quantized cache to fit within the 4090’s memory. I ran it through a local API and measured both token generation and the wait for a complete response. I started in LM Studio, where Qwen3.8–27B loaded and answered prompts. Increasing the context setting to accommodate more text led to a sharp slowdown. The larger setting left little memory free, so I first reduced the footprint rather than continuing to increase capacity. The configuration used for the speed test was: The file labels describe quantized representations. They store numerical weights more compactly, making the model easier to fit in memory, with a possible effect on answer quality. The KV cache is separate: it stores information from processed text so the model can reuse it while generating an answer. That distinction matters during setup. A model file can fit while leaving too little space for the cache and runtime. In my case, moving to smaller weights and a quantized cache made room for the rest of the configuration. The runtime changed along with the memory settings. Once the server is running, the main model has help generating text. A small model proposes a block of upcoming tokens, and the main model verifies it. Accepted proposals let generation advance several tokens in one round. This is speculative decoding . The official diagram shows how DFlash uses features from the target model to guide a lightweight diffusion drafter. The drafter predicts several tokens in parallel. DFlash2 https://inco.ai/blog/dflash2/ builds on this with local convolutions and a candidate selector to improve the proposed block. Rejected proposals still consume time, and the drafter itself occupies GPU memory. I used four as the maximum draft length after earlier comparisons suggested that short blocks worked well on my test prompts. It is a starting configuration, not a setting I established as best for every task. The final speed below belongs to the whole setup; I did not run a matched drafting-on versus drafting-off comparison for it. The two files have different jobs: the main model produces the answer, and the matched drafter proposes tokens for it to check. Download Qwen3.8-27B-UD-Q3 K XL.gguf from the Unsloth file list https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/main , and Qwen3.8-27B-DFlash2-Q4 K M.gguf from the Inco AI GGUF file list https://huggingface.co/incoai/Qwen3.8-27B-DFlash2-GGUF/tree/main . These are current download locations; I have not checked their hashes against my measured copies. For the runtime, follow the llama.cpp Windows and CUDA build instructions https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md . Building on Windows requires the C++ tools, CMake, Git, and CUDA toolkit. The DFlash2 implementation https://github.com/ggml-org/llama.cpp/pull/27342 identifies the required feature. Before loading the models, run llama-server --version and llama-server --help; the help must include draft-dflash and --spec-draft-n-max. An unrecognized option means the executable needs attention before the model settings do. My measured executable reported commit 1deefcc. The command below is its tested launch configuration; a fresh-machine build remains unverified. Run it with llama-server on your PATH, or substitute the full executable path. In PowerShell, a quoted executable path needs & before it. llama-server -m TARGET.gguf -md DRAFT.gguf --alias qwen38 --spec-type draft-dflash --spec-draft-n-max 4 -ngld 99 -ctkd f16 -ctvd f16 -ngl 99 -fa on --jinja -np 1 -c 131072 --cache-type-k q8 0 --cache-type-v q8 0 --host 127.0.0.1 --port 8080 --api-key-file PRIVATE KEY FILE --no-webui TARGET.gguf and DRAFT.gguf are replaced with the downloaded file paths. PRIVATE KEY FILE points to a private text file containing the API key. The server stays in that terminal. A chat client or an agent supporting this API can use http://127.0.0.1:8080/v1, model name qwen38, and the same key. This command starts an API server with its built-in web interface disabled. The model runs locally; prompts sent directly to this endpoint are processed by that server. A first request can come from a second PowerShell terminal, without installing a chat application. Replace PRIVATE KEY FILE with the same key-file path used for the server: php $llmKey = Get-Content -Raw -LiteralPath 'PRIVATE KEY FILE' .Trim $llmBody = @{ model = 'qwen38' messages = @ @{role = 'user'; content = 'Explain GPU memory in two sentences.'} max tokens = 512 temperature = 0 stream = $false} | ConvertTo-Json -Depth 5$llmReply = Invoke-RestMethod -Method Post -Uri 'http://127.0.0.1:8080/v1/chat/completions' -Headers @{Authorization = "Bearer $llmKey"} -ContentType 'application/json' -Body $llmBody$llmReply.choices 0 .message.content$llmReply.choices 0 .finish reason An answer in message.content confirms the request reached the model. stop means generation ended normally; length means the output budget cut it off. A 401 response points to the key, while a connection failure calls for checking the server terminal and port. This connection check uses a different prompt and output budget from the timing tests below. The settings worth recognizing are the draft length of four, the quantized main-model cache, and one concurrent request. The 131072 value sets context capacity. It is the capacity I used, not a requirement for short conversations. A smaller context is a sensible starting point when the task needs less or memory is tight. In the September 15 measurement, I checked this configuration with two short prompts in one server session. Each received a warmup followed by three measured requests, using greedy decoding and cached-prefix reuse. TaskGeneration speedFull response timeOutputShort prose~100 tok/s1.68 s157 tokens; completedCode request~103 tok/s2.57 s250 tokens; cut off Averages from three measured requests per prompt, after one warmup each, in one server session. The prose request asked for a three-sentence explanation of linear attention and the KV cache. Its generation took about 1.56 seconds, so the full response followed shortly afterward. The code request asked for a pandas function to clean and group a CSV. It reached the output limit every time; its row measures speed, not a completed coding task. Tokens are pieces of text, so these rates are not words per second. Different languages and answers can produce different token counts. The distinction is most visible with a long input. The model first processes the prompt — prefill — then generates the answer — decode . A fast decode rate cannot tell me how long a new document will take to process. My actual prompts were 66 and 97 tokens long. Although the server had 128K capacity, these were short, cached requests. They show that this configuration can respond quickly under those conditions. They do not establish the same waiting time for a large document or several users at once. The RTX 4090 reached roughly 100 tokens per second with this Qwen3.8–27B configuration on short, cached requests. The prose response completed in an average of 1.68 seconds. The model could also receive requests through a local API — the interface I wanted for an agent running on my PC. This gives me a working model server with a measured response time. Agent task completion and all-day reliability require their own evaluation; neither was measured in this speed test. The question I started with now has an answer: under these conditions, my 4090 can run this 27B model at about 100 tokens per second. I have the launch settings and API connection to use that result. For the local agent I want to build, this is a concrete candidate with its first requirement checked: fast generation on the hardware already on my desk. Running Qwen3.8–27B on an RTX 4090 https://pub.towardsai.net/running-qwen3-8-27b-on-an-rtx-4090-cde0cb4ba338 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.