DeepSeek V4 Flash now runs from a backpack #
A community tester has run the new DeepSeek-V4-Flash-0731 quant on a Bosgame M5 mini PC — a small-form-factor box with an RTX PRO 6000 Max-Q external GPU (an eGPU, a graphics card in a Thunderbolt enclosure rather than a motherboard slot) bolted on. The result: a frontier-class open-weight model serving 44 to 60 tokens per second depending on which quantisation you pick, with no hyperscaler in the loop. DeepSeek V4 Flash — a large open-weight AI model that’s been making waves as a cheaper alternative to the leading US frontier — has just become something you could carry to a client meeting.
The numbers come from a hands-on test posted in the same week as an NVIDIA Developer Forums thread documenting the same model serving on DGX Spark dev kits. One of the forum testers, jasl, summed up the bottleneck in two words: Decode is memory bandwidth-limited.
The portable rig uses llama.cpp, an open-source local-inference engine, with a custom GGUF build (a compact file format tuned for CPU-friendly inference), splitting model layers across the eGPU and the integrated GPU when the model outgrew the eGPU’s memory. A speculative-decoding helper — a small companion model that guesses tokens ahead so the main model only checks them — was ported from a closed PR and added for the larger builds.
A bug in an alternative inference engine — a local-serving alternative to llama.cpp — returns all-zero output tokens on Nvidia’s current-generation GPUs. This matters if you were planning to use that engine for this model.
Three sizes, three speeds #
The test covered three quantisation builds, each a different balance of quality against footprint:
Smallest build— fits the 24 GB eGPU cleanly and runs fastest; the entire model sits on the eGPU with no helper.** Mid build**— split across the eGPU and the integrated GPU plus the drafter; quality improves, decode drops modestly.** Largest build**— same split, slightly more on the integrated GPU; highest quality, slowest decode of the three.
59.5 t/sdecode on the 90 GiB Q2_K_XL build, with the whole model on the RTX PRO 6000 Max-Q eGPU
The smallest build is the one to read twice. It loses some quality against the larger builds, but it fits the 24 GB eGPU without splitting and runs roughly a third faster. The tester found this the sweet spot for most agentic coding and document work.
The Blackwell caveat #
There’s a related bug worth knowing about. The same issue reports that DeepSeek V4 Flash returns all-zero output tokens on a single Blackwell card under that engine’s path, and crashes with a floating-point error at higher temperatures. The bug doesn’t affect the llama.cpp route the tester used, so the portable rig is unaffected — but if you were planning to use that engine for higher-throughput prefill or the official 4-bit expert path on Blackwell, you’ll be waiting on a fix (issue #2001).
How to try this yourself #
If you’ve already got a Thunderbolt-equipped mini PC and a Blackwell eGPU, here’s the weekend-sized plan: Start at the smallest quantisation, not the largest. The 90 GiB build runs entirely on a 24 GB eGPU and gives the best decode numbers. Move up only if you can measure a quality gap you care about; thellama.cpp benchmark piecefound llama.cpp sits within roughly 6% of vLLM on most tasks.Stay on llama.cpp until the fix lands. The alternative inference engine returns all-zero output on current-generation Nvidia cards untilissue #2001is resolved. llama.cpp with the UD-Q*_K_XL quants is the working route.Port the speculative drafter if you can. Speculative decoding is what lifts the larger builds from “uncomfortable” to “actually responsive.” The tester reports a draft-acceptance rate around 53%, which roughly halves effective latency on those builds.Watch memory bandwidth, not just VRAM. Decode on small boxes is bandwidth-bound; a 24 GB eGPU over PCIe 5.0 x4 will outrun one over PCIe 4.0 x4 at the same VRAM size. The DGX Spark numbers in the same forum thread tell the same story.For most UK small teams: rent the model, don’t carry it. This rig is a useful proof point for one-off sensitive workloads where data must not leave the building, but expensive per seat compared with API access. The portable-rig pattern is what to watch, not today’s specific build — seepicking your first AI team planfor the cost-vs-control trade-off in pounds.
If you don’t have the kit, the same model running on a single DGX Spark is documented in the NVIDIA forum thread with working vLLM recipes — worth reading before you spend money on hardware.
Sources & quotes #
Every quotation in this article is verbatim from a named source — click any 1 to see where it came from. It's part of how we keep an AI-run newsroom honest. How we verify →