{"slug": "running-qwen3-8-flash-next-nvfp4-at-184-tok-s-on-a-single-rtx-6000-pro", "title": "Running Qwen3.8-Flash-Next NVFP4 at 184 tok/s on a single RTX 6000 Pro", "summary": "Mesh Applied released two quantized open models on Hugging Face: Goethe Qwen3.8-27B NVFP4 NInfer, which runs on a 32GB VRAM card such as the RTX 5090 (and on 24GB) at roughly 255 tok/sec at concurrency 1 and about 686 tok/sec at concurrency 4, and Goethe Qwen3.8-Flash-Next NVFP4, which the company reports running at 184 tok/sec on a single RTX 6000 Pro. Mesh Applied said open models are already good enough for about 98% of the tasks people use AI for, and it frames its work as shortening the \"joule to prediction\" chain by pairing post-training, quantization and purpose-built kernels with widely available hardware.", "body_md": "# Our first model-kernel combos\n\n### **The joule-to-prediction supply chain**\n\nThe world is rearranging itself around compute. If you can throw enough reliable prediction at a problem, a surprising fraction of it becomes solvable. Some of those problems have immediate financial incentives baked in (coding, trading, cyber), while others don’t (basic science, public infrastructure, education, climate). The incentive gradient is uneven, but the method is the same: turn energy into predictions that change the next action.\n\nOpen models are already good enough for something like 98% of the tasks people actually use AI for; drafting, search-shaped reasoning, glue code, triage, tutoring, light agents. The frontier still matters, but for most work the bottleneck is that can I run a good enough model frequently and fast?\n\nIf energy becomes abundant, cost collapses toward a function of time and waste. A task still finishes in finite time, and that time is what you feel. Time per task becomes the primary metric. We think the supply chain that matters most is the joule to prediction chain. Everything we ship is an attempt to make that chain shorter and leaner.\n\n### **Our first models:**\n\nOur philosophy behind model creation is to find widespread hardware setups with the least amount of unnecessary bottlenecks (such as PCIe limits or kernel inaccessibility), then calculate what’s the largest model we can run on them after a series of post training, quantisation and kernel optimisation stages. The goal is to achieve the highest possible throughput (measured in tok/sec), while retaining model capability on topics that matter. This results in high performance purpose built models, that cost much less per task than the base model.\n\n#### [🤗](https://huggingface.co/meshapplied/Qwen3.8-27B-NVFP4-NInfer) Goethe Qwen3.8-27B NVFP4 NInfer\n\nFind the model here: [https://huggingface.co/meshapplied/Qwen3.8-27B-NVFP4-NInfer](https://huggingface.co/meshapplied/Qwen3.8-27B-NVFP4-NInfer)\n\nWhile large MoE models are extremely good at reasoning and deep domain specific knowledge, sometimes a dense model is preferred. For standard RAG/personal agents or even small coding tasks, time/task is more important than using a highly capable model. Dense models are more straightforward to compress, because every single weight is quantised. With a good compression algorithm that retains the capability, and a purpose built kernel, we can reduce the time/task to a few seconds from minutes.\n\nThis model is designed to run on consumer hardware, so people can task their own proprietary agents from anywhere and get a response immediately. For this model we recommend a using a 32GB vram card (ex.: RTX5090), but the model runs on 24GB VRAM as well. We didn’t do cost estimations because it’s not designed for a sustained load.\n\nThe model achieves ~255 tok/sec at concurrency = 1, and a whopping ~686 tok/sec at concurrency = 4. You won’t find an analytical solution to the Navier-Stokes with it, but querying documents, answering questions, studying, searching the web; this might just be the best friend you ever had…\n\n*(Performance metrics were measured on a single GeForce RTX5090 32GB card on heldout agentic coding.)*\n\n#### [🤗](https://huggingface.co/meshapplied/Qwen3.8-Flash-Next-NVFP4-Latest) Goethe Qwen3.8-Flash-Next NVFP4\n\nFind the model here: [https://huggingface.co/meshapplied/Qwen3.8-Flash-Next-NVFP4-Latest](https://huggingface.co/meshapplied/Qwen3.8-Flash-Next-NVFP4-Latest)\n\nThe first task we aim to cover is the most widespread task there is: coding. Coding is not just returning text for a given input. It’s a complex task which requires an agent to read and write into multiple directories, understand relationships, derive conclusions, and rely on not just the context it is given, but also the colorful knowledge hidden in it’s weights. There are many levels of agentic coding of course, but in most of the cases it’s pretty standard problem solving.\n\nThe model’s size and desired operation (large context, long runtime) made us choose a 96GB VRAM setup. This accommodates model weights, leaves plenty room to KV cache, and supports concurrent requests (subagents/multiple users). We recommend using a single 96GB VRAM card (ex.: RTX6000 PRO), this unlocks the best performance possible, but 2x 48GB cards can also give acceptable performance. In a self-hosted setup the serving cost per 1M output tokens comes out to less than 0.01 USD.\n\nIn performance, the model reliably brings [~184 tok/s @ concurrency = 1, ~334 tok/s @ concurrency = 4] on decode, while slightly outperforming the capabilities of Gemini 3.8-Flash, Claude Opus-4.6 in math and coding.\n\n*(Performance metrics were measured on a single RTX6000 Pro MaxQ 96GB card on heldout agentic coding.)*\n\nKernel extension and framework on [Github](https://github.com/meshapplied/goethe).", "url": "https://wpnews.pro/news/running-qwen3-8-flash-next-nvfp4-at-184-tok-s-on-a-single-rtx-6000-pro", "canonical_source": "https://meshapplied.com/blog/release-1", "published_at": "2026-10-26 22:17:22+00:00", "updated_at": "2026-10-26 22:41:44.872410+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "ai-chips"], "entities": ["Mesh Applied", "Qwen3.8-27B NVFP4 NInfer", "Qwen3.8-Flash-Next NVFP4", "Hugging Face", "RTX 6000 Pro", "RTX 5090"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/running-qwen3-8-flash-next-nvfp4-at-184-tok-s-on-a-single-rtx-6000-pro", "markdown": "https://wpnews.pro/news/running-qwen3-8-flash-next-nvfp4-at-184-tok-s-on-a-single-rtx-6000-pro.md", "text": "https://wpnews.pro/news/running-qwen3-8-flash-next-nvfp4-at-184-tok-s-on-a-single-rtx-6000-pro.txt", "jsonld": "https://wpnews.pro/news/running-qwen3-8-flash-next-nvfp4-at-184-tok-s-on-a-single-rtx-6000-pro.jsonld"}}