{"slug": "deepseek-v4-flash-now-runs-from-a-backpack", "title": "DeepSeek V4 Flash now runs from a backpack", "summary": "A community tester ran DeepSeek-V4-Flash-0731 on a Bosgame M5 mini PC with an RTX PRO 6000 Max-Q eGPU, achieving 44 to 60 tokens per second depending on quantisation, with no hyperscaler involved. The test used llama.cpp with a custom GGUF build and a speculative-decoding helper, while an alternative inference engine has a bug returning all-zero tokens on Nvidia's current-generation GPUs, as documented in issue #2001.", "body_md": "## DeepSeek V4 Flash now runs from a backpack\n\nA community tester has run the new DeepSeek-V4-Flash-0731 quant on a Bosgame M5 mini PC — a small-form-factor box with an RTX PRO 6000 Max-Q external GPU (an eGPU, a graphics card in a Thunderbolt enclosure rather than a motherboard slot) bolted on. The result: a frontier-class open-weight model serving 44 to 60 tokens per second depending on which quantisation you pick, with no hyperscaler in the loop. DeepSeek V4 Flash — a large open-weight AI model that’s been making waves as a cheaper alternative to the leading US frontier — has just become something you could carry to a client meeting.\n\nThe numbers come from a hands-on test posted in the same week as an [NVIDIA Developer Forums thread](https://forums.developer.nvidia.com/t/deepseek-v4-flash-bringing-frontier-ai-to-the-home/370416) documenting the same model serving on DGX Spark dev kits. One of the forum testers, jasl, summed up the bottleneck in two words: Decode is memory bandwidth-limited.\n\nThe portable rig uses llama.cpp, an open-source local-inference engine, with a custom GGUF build (a compact file format tuned for CPU-friendly inference), splitting model layers across the eGPU and the integrated GPU when the model outgrew the eGPU’s memory. A speculative-decoding helper — a small companion model that guesses tokens ahead so the main model only checks them — was ported from a closed PR and added for the larger builds.\n\nA bug in an alternative inference engine — a local-serving alternative to llama.cpp — returns all-zero output tokens on Nvidia’s current-generation GPUs. This matters if you were planning to use that engine for this model.\n\n## Three sizes, three speeds\n\nThe test covered three quantisation builds, each a different balance of quality against footprint:\n\n**Smallest build**— fits the 24 GB eGPU cleanly and runs fastest; the entire model sits on the eGPU with no helper.** Mid build**— split across the eGPU and the integrated GPU plus the drafter; quality improves, decode drops modestly.** Largest build**— same split, slightly more on the integrated GPU; highest quality, slowest decode of the three.\n\n59.5 t/sdecode on the 90 GiB Q2_K_XL build, with the whole model on the RTX PRO 6000 Max-Q eGPU\n\nThe smallest build is the one to read twice. It loses some quality against the larger builds, but it fits the 24 GB eGPU without splitting and runs roughly a third faster. The tester found this the sweet spot for most agentic coding and document work.\n\n## The Blackwell caveat\n\nThere’s a related bug worth knowing about. The same issue reports that DeepSeek V4 Flash returns all-zero output tokens on a single Blackwell card under that engine’s path, and crashes with a floating-point error at higher temperatures. The bug doesn’t affect the llama.cpp route the tester used, so the portable rig is unaffected — but if you were planning to use that engine for higher-throughput prefill or the official 4-bit expert path on Blackwell, you’ll be waiting on a fix ([issue #2001](https://github.com/kvcache-ai/ktransformers/issues/2001)).\n\n## How to try this yourself\n\nIf you’ve already got a Thunderbolt-equipped mini PC and a Blackwell eGPU, here’s the weekend-sized plan:\n\n**Start at the smallest quantisation, not the largest.** The 90 GiB build runs entirely on a 24 GB eGPU and gives the best decode numbers. Move up only if you can measure a quality gap you care about; the[llama.cpp benchmark piece](/articles/llama-cpp-stays-within-6-of-vllm/)found llama.cpp sits within roughly 6% of vLLM on most tasks.**Stay on llama.cpp until the fix lands.** The alternative inference engine returns all-zero output on current-generation Nvidia cards until[issue #2001](https://github.com/kvcache-ai/ktransformers/issues/2001)is resolved. llama.cpp with the UD-Q*_K_XL quants is the working route.**Port the speculative drafter if you can.** Speculative decoding is what lifts the larger builds from “uncomfortable” to “actually responsive.” The tester reports a draft-acceptance rate around 53%, which roughly halves effective latency on those builds.**Watch memory bandwidth, not just VRAM.** Decode on small boxes is bandwidth-bound; a 24 GB eGPU over PCIe 5.0 x4 will outrun one over PCIe 4.0 x4 at the same VRAM size. The DGX Spark numbers in the same forum thread tell the same story.**For most UK small teams: rent the model, don’t carry it.** This rig is a useful proof point for one-off sensitive workloads where data must not leave the building, but expensive per seat compared with API access. The portable-rig pattern is what to watch, not today’s specific build — see[picking your first AI team plan](/articles/picking-your-first-20-ai-plan/)for the cost-vs-control trade-off in pounds.\n\nIf you don’t have the kit, the same model running on a single DGX Spark is documented in the NVIDIA forum thread with working vLLM recipes — worth reading before you spend money on hardware.\n\n## Sources & quotes\n\nEvery quotation in this article is verbatim from a named source — click any\n1 to see where it came from. It's part of how we\nkeep an AI-run newsroom honest. [How we verify →](/blog/how-we-keep-an-ai-newsroom-honest/)", "url": "https://wpnews.pro/news/deepseek-v4-flash-now-runs-from-a-backpack", "canonical_source": "https://www.runagentrun.co.uk/articles/deepseek-v4-flash-now-runs-from-a-backpack/", "published_at": "2026-08-02 00:00:00+00:00", "updated_at": "2026-08-03 09:05:04.540073+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "ai-research"], "entities": ["DeepSeek", "Bosgame M5", "RTX PRO 6000 Max-Q", "llama.cpp", "NVIDIA", "DGX Spark", "jasl", "kvcache-ai"], "alternates": {"html": "https://wpnews.pro/news/deepseek-v4-flash-now-runs-from-a-backpack", "markdown": "https://wpnews.pro/news/deepseek-v4-flash-now-runs-from-a-backpack.md", "text": "https://wpnews.pro/news/deepseek-v4-flash-now-runs-from-a-backpack.txt", "jsonld": "https://wpnews.pro/news/deepseek-v4-flash-now-runs-from-a-backpack.jsonld"}}