{"slug": "running-qwen3-8-flash-on-an-rtx-4090-would-i-replace-my-27b-model", "title": "Running Qwen3.8 Flash on an RTX 4090: Would I Replace My 27B Model?", "summary": "A hands-on test by Kwangju Shin (KJ) found that Qwen3.8 Flash running on an RTX 4090 via the open-source Strata inference engine decoded at about 126 tokens per second in English and 118 in Korean with IQ3_XXS weights, versus roughly 80 and 50 tokens per second for his Qwen3.8-27B setup with Q4 weights, DFlash2 and a 32K context. The advantage reversed on document tasks: the 27B model answered a fresh document query in 5.69 seconds while IQ3_XXS took 16.93 seconds, and IQ3_XXS left only about 2GiB of system RAM free on his 64GB desktop. Shin cautioned that the 512-token generation test and a single document request were too little to establish a reliable speed ratio, making him wary of choosing a model from tokens-per-second figures alone.", "body_md": "*Kwangju Shin (KJ)*\n\nIn my [previous article](https://medium.com/towards-artificial-intelligence/running-qwen3-8-27b-on-an-rtx-4090-cde0cb4ba338), I got Qwen3.8–27B running at roughly 100 tokens per second on short, cached requests. Once that worked, I wanted to try Qwen3.8 Flash on the same RTX 4090. Could it give my local agent better answers without a longer wait?\n\nFlash generated text faster in this comparison. The version I focused on, IQ3_XXS, left only about 2GiB of system RAM available on my 64GB desktop.\n\nI use this PC for other work too. A model waiting for the agent’s next request would have to share it with me. That made the memory left over as interesting as the generation speed.\n\nTo try Flash, I needed software that could run it across the hardware I already owned. I used [Strata](https://github.com/Niko1221/Strata), an open-source inference engine with a local, OpenAI-compatible API. Qwen3.8-Flash-Next is the model; Strata runs it and lets a program send it requests.\n\nBut how does a model too large for the GPU run on this PC at all?\n\nFlash uses many small specialist networks, called experts. For each token, only a subset participates in the calculation. Strata takes advantage of that selective use. Its official diagram shows how the work is divided:\n\nThe GPU runs the model’s core computation and keeps frequently used experts close by. RAM holds the expert weights; when a required expert is absent from the GPU, the CPU can calculate its contribution. The SSD has another job in this design: it holds a large lookup table, from which only needed rows are read. That table does not have to remain entirely in RAM.\n\nThat made Flash possible to try on this PC. It also made the rest of the machine part of the experiment: how much RAM would be left, and how long would I wait for an answer?\n\nMy PC had an RTX 4090 with 24GB of VRAM, 64GB of system RAM, and Windows. I ran one local server at a time and sent the same requests to a remote API as a reference.\n\nI tried three compressed versions of the model weights: IQ2_XS, IQ3_XXS, and IQ3_S. This compression, called quantization, saves space but can affect the answers. IQ3_XXS became the main candidate I compared with 27B; the other two helped show what changed with compression.\n\nFor this comparison, I ran 27B with Q4 weights, DFlash2, and a 32K context setting. The previous article used Q3 weights and short, cached inputs, so I measured 27B again alongside Flash rather than reusing its earlier 100-token result.\n\nFlash also had help generating text. Its multi-token prediction (MTP) drafting proposed upcoming tokens for the model to check. As with DFlash2 in the first article, the speed belongs to the whole running setup.\n\nWith reasoning mode disabled, the server-reported decode rates for IQ3_XXS were about 126 tokens per second in English and 118 in Korean. The 27B setup reached about 80 and 50 respectively. These rates measure generation; the waiting times below also include input processing and response overhead.\n\nThe waiting time made that difference easier to understand:\n\nThat was enough to make Flash worth considering for tasks with more output. The more compressed IQ2_XS version was faster still. This test stopped at 512 tokens, though; I had not established how the advantage would hold over a much longer answer.\n\nBut I also wanted to send it documents. That changed the comparison.\n\nI sent a fresh document and asked for an answer from it. All five configurations answered correctly. The 27B model returned its answer in 5.69 seconds. IQ3_XXS took 16.93.\n\nBefore generating an answer, a model has to process what I send it. In the previous article, I separated input processing (prefill) from output generation (decode). That distinction mattered here: Flash won the generation test, but 27B finished this document task sooner.\n\nOne document request is too little to establish a reliable speed ratio. It was enough to make me wary of choosing a model from its tokens-per-second number alone. For an agent that receives documents or tool results, the input is part of every request too.\n\nMy test questions included calculations, extracting information from documents, scheduling, and small optimization problems. The fastest Flash version, IQ2_XS, repeatedly got a resource-allocation total wrong and made an error in a pump calculation. That would leave me checking and correcting the output I had received so quickly.\n\nIQ3_XXS looked more useful. After excluding one ambiguous question, every answer it finished was correct. Five times, though, it ran out of output allowance before reaching a final answer. The 27B model also got every completed answer right and left four unfinished. On this adjusted set, the two were close.\n\nThe ambiguous question was my mistake: I had asked for an order quantity without saying whether I wanted packs or individual items. Several models returned seven packs where my expected answer was 168 items. I corrected the wording in a follow-up, and all five got it right. A score can hide a problem in the question as easily as a problem in the model.\n\nIQ3_S also gave me a reason to keep it in mind. When I increased the output allowance from 8K to 16K for two difficult optimization problems, it solved one that the other local configurations still failed to finish. This was a single follow-up per problem, so I would treat it as something to investigate further. The remote API solved both.\n\nThis small suite did not give me a clear answer-quality reason to replace 27B with IQ3_XXS.\n\nThe part that mattered most for my desktop was how little RAM remained available.\n\nThe 27B setup left about 21GiB available. IQ2_XS left about 9.3GiB. IQ3_XXS left 2.24GiB, and IQ3_S left less than 1GiB at the median.\n\nFlash could finish the benchmark. Keeping it running while I continued working was another requirement. With IQ3_S, available RAM fell below 1GiB in 339 of 653 samples. That gave me very little room to add anything else.\n\nThe available RAM was measured with my other applications retained, not on an otherwise empty machine. It tells me about memory headroom. It does not establish how either setup behaves alongside another GPU-heavy program, through a full working day, or as an agent accumulates a longer conversation.\n\nFor now, I would start with the 27B Q4 + DFlash2 setup tested here. It generated text at a useful speed, returned the fresh-document answer sooner, and left about 21GiB of RAM available. The clearer improvement in answers I had hoped for from IQ3_XXS did not appear in this small test.\n\nFlash IQ3_XXS remains worth trying when I can devote more of the PC to a request. Its Korean output speed gives me a reason to test longer generation jobs. I would be less comfortable keeping it in the background with only 2.24GiB of RAM to spare.\n\nNeither setup has been validated for an agent running all day. This is the configuration I would start that work with, not a finished agent system.\n\nThe first article gave me a local model my agent could call. This comparison gave me a way to choose what stays available: send it a fresh document, check the answers, and see what room it leaves for the rest of the PC. For mine, that points to 27B for now. I want the model ready for the next request while I get on with my work.\n\n[Running Qwen3.8 Flash on an RTX 4090: Would I Replace My 27B Model?](https://pub.towardsai.net/running-qwen3-8-flash-on-an-rtx-4090-would-i-replace-my-27b-model-f8d40a54b575) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/running-qwen3-8-flash-on-an-rtx-4090-would-i-replace-my-27b-model", "canonical_source": "https://pub.towardsai.net/running-qwen3-8-flash-on-an-rtx-4090-would-i-replace-my-27b-model-f8d40a54b575?source=rss----98111c9905da---4", "published_at": "2026-11-05 15:01:04+00:00", "updated_at": "2026-11-05 15:29:31.579330+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "ai-research"], "entities": ["Qwen3.8 Flash", "Qwen3.8-27B", "Strata", "RTX 4090", "Kwangju Shin", "Qwen3.8-Flash-Next", "IQ3_XXS", "IQ2_XS"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/running-qwen3-8-flash-on-an-rtx-4090-would-i-replace-my-27b-model", "markdown": "https://wpnews.pro/news/running-qwen3-8-flash-on-an-rtx-4090-would-i-replace-my-27b-model.md", "text": "https://wpnews.pro/news/running-qwen3-8-flash-on-an-rtx-4090-would-i-replace-my-27b-model.txt", "jsonld": "https://wpnews.pro/news/running-qwen3-8-flash-on-an-rtx-4090-would-i-replace-my-27b-model.jsonld"}}