Dual Sparks in nvfp4 vs 4x RTX Pro 6000 with native DeepSeek V4 0731 -- Quants and Speed

A homelab user deploying DeepSeek-V4-Flash-0731 on two DGX Spark units achieved 1M token context with an NVFP4 KV cache using a custom vLLM fork, reporting a KV cache size of 1,492,347 tokens and maximum concurrency of 1.42x per request. The deployment required a specific configuration including tensor parallelism of 2, dual ConnectX-7 200 Gb/s RoCE fabric, and a custom vLLM 0.21.1rc1.dev339+g1967a5627bc3 runtime, with the model served at 166.9 GB across 48 safetensors shards. The author notes that stock vLLM versions 0.24 and 0.25.2 are incompatible and that deviations from the recommended settings (MAX_NUM_SEQS=6, GPU_MEMORY_UTILIZATION=0.78, and 5 speculative tokens) cause crashes.

DeepSeek-V4-Flash-0731 on 2x DGX Spark: 1M Context, NVFP4 KV Cache A practical, field-tested companion to the upstream guide: GitHub - tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark: DeepSeek V4 Flash DSpark 1M NVFP4 KV recipe for 2x DGX Spark · GitHub https://github.com/tonyd2wild/DeepSeek-V4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark This document is written for a homelab enthusiast who already owns two DGX Spark units GB10, SM120 , has a ConnectX/QSFP fabric between them, runs Docker, and wants a single 304B-parameter model serving at 1M context with an NVFP4 KV cache. It is a companion, not a replacement. Read the guide first, then use this to avoid the footguns we hit on a real deployment. Everything below was verified on our own pair of Sparks, not copied from the README. IMPORTANT This recipe runs a custom vLLM fork , not stock vLLM. Do not point stock vLLM main, 0.24, or 0.25.2 at this model on DGX Spark. See Why not stock vLLM why-not-stock-vllm below. The shape of the thing | Component | Value | | Hardware | 2x DGX Spark GB10, SM120 , 1 GPU per node | | Tensor parallelism | TP=2 one GPU per node | | Fabric | Dual ConnectX-7 200 Gb/s RoCE RoCEv2, GID index 3 | | Base image | ghcr.io/bjk110/vllm-spark:unholy-fusion-prod-ready public, no auth, ~21.2 GiB | | Runtime | vLLM 0.21.1rc1.dev339+g1967a5627bc3, FlashInfer 0.6.12, TileLang 0.1.9, Python 3.12 at /opt/env | | Model | deepseek-ai/DeepSeek-V4-Flash-0731 , rev 7872f01b , 166.9 GB, 48 safetensors shards | | KV cache dtype | nvfp4 ds mla the “NVFP4” in the repo name is the KV cache, not the weights | | Context | 1,048,576 tokens YaRN factor 16 over 65,536; 1M is the calibrated ceiling | | API | http://<head-fabric-ip :8888/v1 | The build produces four images per node overlay, stage-a, stage-b, stage-c ; stage-c is canonical and is what you run. Key .env.dspark values from our working config WORKER HOST=<worker hostname or IP MASTER ADDR=10.10.10.1 MASTER PORT=25440 VLLM HOST IP=10.10.10.1 WORKER VLLM HOST IP=10.10.10.2 NCCL IB HCA=rocep1s0f1,roceP2p1s0f1 NCCL SOCKET IFNAME=enp1s0f1np1 NCCL IB GID INDEX=3 HF CACHE=/home/spark2/.cache/huggingface DSPARK MODEL=deepseek-ai/DeepSeek-V4-Flash-0731 Our fabric used 10.10.10.1/.2 ; the guide uses 192.168.192.0/24 . Your subnet and device names will differ. The values above are the ones that worked for us. Default config current best per the guide, and what we ran : MAX MODEL LEN=1048576 MAX NUM SEQS=6 GPU MEMORY UTILIZATION=0.78 MTP NUM TOKENS=5 kv-cache-dtype nvfp4 ds mla speculative config {"method":"dspark","num speculative tokens":5,"draft sample method":"probabilistic"} --distributed-executor-backend mp Deviating from MAX NUM SEQS=6 , GPU MEMORY UTILIZATION=0.78 , or k=5 reproduces documented crashes. The model card recommends num speculative tokens=7 , but this drafter’s dspark block size is 5 and k=7 does not work on it. Use 5. Launch sequence ./build-dspark-vllm-runtime.sh build overlay + stage A/B/C on head AND worker ./prepare-dspark-model-cache.sh snapshot download + verify + mirror to worker ./start-deepseek-v4-flash-dspark.sh worker-first launch + smoke test The API lands at http://<head-fabric-ip :8888/v1 . Boot markers to look for: GPU KV cache size 1,492,347 tokens Maximum concurrency for 1,048,576 tokens per request: 1.42x Application startup complete. Then the test: chat with "Reply exactly: NVFP4 DSPARK OK" should return exactly that string. Where the guide needs adjustment we hit all of these The guide is good, but it assumes a specific lab. On your hardware and setup, you will need to fix these. 1. Baked-in VLLM HOST IP footgun fatal if ignored The base image has VLLM HOST IP=192.168.192.1 baked in. On any node whose fabric IP is not 192.168.192.1 , the worker dies immediately with: zmq.error.ZMQError: Cannot assign requested address Our image also carried a stale NODE IP=192.168.50.58 . Fix: set VLLM HOST IP head and WORKER VLLM HOST IP worker explicitly per node, and override the stale NODE IP . Do not rely on anything baked into the image. 2. NCCL IB HCA is per-node, not uniform The image bakes NCCL IB HCA=rocep1s0f0 , but RoCE device names are not uniform across units. Auto-detect the Up RoCE device whose netdev carries the fabric IP. On both of our nodes that resolved to: NCCL IB HCA=rocep1s0f1,roceP2p1s0f1 NCCL IB GID INDEX=3 RoCEv2 Verify with ibstat / ibv devinfo rather than trusting the baked value. 3. The model must be on BOTH nodes TP=2 gotcha With TP=2 every node needs the full 166.9 GB of weights. Download once to NFS or the head , then rsync to both nodes’ local HF caches. With HF HUB OFFLINE=1 the runtime resolves the model fully offline, which is how you avoid a surprise 167 GB re-download at boot. 4. Leftover DSPARK MODEL string .env.dspark.example defaults DSPARK MODEL to deepseek-ai/DeepSeek-V4-Flash-DSpark , which is a leftover name. The real HF repo is deepseek-ai/DeepSeek-V4-Flash-0731 . Point it at the 0731 local cache path and run with HF HUB OFFLINE=1 . 5. capture runtime.sh container-name mismatch The script’s default CONTAINER NAME is the title-cased basename of the directory, which does not match the lowercase deploy container name. If you do not pass it explicitly, the captured files come back empty. Pass: CONTAINER NAME=deepseek-v4-flash-0731-dspark-1m-nvfp4-kv-2x-dgx-spark-vllm-dspark-1 6. Reasoning-mode gotcha on the full endpoint This only matters if you also run the 4x RTX6000 comparison see below , but it is worth knowing: the “full” stock-vLLM deployment runs in REASONING mode. It emits a separate reasoning field before content , and small max tokens budgets get eaten by reasoning. At max tokens=16 you get content: null ; at 64 you get "I am DeepSe" . Use max tokens = 1024 , or hit the raw /v1/completions path, which skips reasoning entirely. 7. First boot is slow. Do not panic. First boot takes on the order of 281 s: model load ~79.51 GiB , DeepGEMM warmup, FlashInfer SM120 sparse-MLA autotune, and torch.compile . Later boots reuse caches and are much faster. And before you trust any benchmark number, send real traffic first: cold runs are ~30% low until the JIT kernels warm up. 8. Repo rename The repo was renamed from DeepSeek-v4-Flash-... to DeepSeek-v4-Flash-... a case change . The old -v4- URL still works, but use the current one above. Why not stock vLLM This is the single most important thing to understand. Stock vLLM cannot serve this model on DGX Spark. The guide documents this in UPSTREAM V024 STATUS.md : a 2026-07-01 test of vLLM main which already contained the merged DSpark PR imported the DSpark modules but did not boot the checkpoint on 2x DGX Spark/SM120. The failures were backend-level: Unknown SF transformation, a Marlin FP4 repack PTX error, the FlashInfer backend rejecting the device, and MXFP4 method incompatibility. A direct bake-off in RUNTIME-BAKEOFF-2026-07-29.md against vLLM 0.25.2 ghcr.io/anemll/dspark-vllm-gx10:0.1.1 confirmed the gap: stock lost 9% on peak, 8% on mean, and 29% at concurrency c6. The gap is step time from the B12X MoE kernels plus a working torch.compile path, both of which only exist on the old fork. The out-of-tree patch inventory one line each | Patch | What it does | B12X MoE kernels b12x moe.py , VLLM USE B12X MOE | The fused MoE kernels that make decode fast on SM120 | B12X MLA sparse attention b12x mla sparse.py | Sparse-MLA attention for the V4 architecture | DeepSeekV4 model support models/deepseek v4/ | Model implementation for SM120/GB10 | Spec-decode proposer Patches 1/2/2b dspark proposer.py , dspark.py , gpu model runner.py | The MTP drafter and its plumbing | Scheduler Patch 3 scheduler.py | No spec placeholders on prefill chunks | NVFP4 envelope Stage A/B/C nvfp4 ds mla dtype | 416-byte probe validated to a 584-byte padded envelope | Patch 4 0731, STACKED PARAM NAME MAPPING | Adds shared experts.gate up proj rows; fixes 12 dropped tensors | keys-concurrency.patch | External vendored patch | official-main-b12x-nvfp4-python.patch | Experimental, not the default | Patch 4 is the big one for 0731. It fixes 12 dropped tensors and takes mean decode from 32.7 to 55.4 tok/s, peak from 42.0 to 66.1, and acceptance from 25.7% to 60.2%. If your acceptance is stuck in the 20-30% range, Patch 4 is missing. Where upstream is heading The upstream vLLM PRs that this fork tracks or parallels: vllm-project/vllm 46995 Spec Decode DSpark merged 2026-07-01 vllm-project/vllm 41834 SM12x support for DeepSeek V4 Flash open vllm-project/vllm 43477 Enable DeepSeek V4 and GLM-5.1 on SM120 merged vllm-project/vllm 40082 FlashInfer B12X MoE and FP4 GEMM for SM120/121 merged vllm-project/vllm 43332 B12X W4A16 merged vllm-project/vllm 47577 Auto-select FLASHINFER B12X NVFP4 on SM120 open vllm-project/vllm 49133 DSpark draft under its own model/quant config open; same failure family as Patch 4 vllm-project/vllm 49617 Fix speculators dspark attribute loading open vllm-project/vllm 48804 Warm spec-decode Triton kernels at startup open vllm-project/vllm 38476 / 47629 TRITON MLA SPARSE backend open vllm-project/vllm 50576 SM8x support open issue vllm-project/vllm 50660 “0731 cannot be stabilized” open issue Repo issues/PRs on the guide itself renamed from -v4- to -v4- ; the old URL still works : 1 perf PR, 2 long-context CUDA assert, 3 garble-fix root cause, 4 long-context crash fix PR, 5 max-cudagraph truncation PR, 6 open soft-failure empty content, 7 proposer fixes via image PR, 8 open engine death CUBLAS/NCCL, 11 multi-node 500 on first request, 12 open “where is the docker image”, 13 open portability PR, 14 open sparkrun recipe PR. The takeaway: this is a moving target. The custom fork is the only thing that works today, upstream is converging several PRs merged , and the open issues above are the ones to watch before you decide to migrate. Model card facts worth knowing deepseek-ai/DeepSeek-V4-Flash-0731 : 304B params, MIT license, text-generation. max position embeddings is 1,048,576 with YaRN factor 16 over 65,536 1M is the calibrated ceiling . Quantization: expert dtype fp4 , weights fp8 e4m3 ue8m0 block 128x128. Config: num key value heads 1 , head dim 512 , q lora rank / o lora rank 1024 , sliding window 128 , index topk 512 , num hash layers 3 . DSpark is attached; the card recommends num speculative tokens 7 , but the config has dspark block size 5 and the guide documents that k=7 does not work on this drafter use 5 . Benchmark table measured vs guide claims, steady state All numbers are tok/s decode and concurrency are aggregate; prefill is TTFT for 1 output token . Measured with stream:false : under spec-decode, streamed deltas measure steps/s, not tokens/s. | Metric | Guide claim | Measured steady-state | Delta | | Decode peak | 84.3 | 84.1 | -0.2% | | Decode mean | 67.6 | 67.5 | -0.1% | | Concurrency c1 | 61.0 | 68.8-69.9 | +13-15% | | Concurrency c2 | 91.7 | 104.6-110.7 | +14-21% | | Concurrency c4 | 151.1 | 163.4-169.0 | +8-12% | | Concurrency c6 | 197.3 | 219.2-226.9 | +11-15% | | Prefill 8K | 1,513 | 1,620 | +7.1% | | Prefill 32K | 2,284 | 2,008 | -12.1% single-shot variance | | Prefill 100K | 2,639 | 2,690 | +1.9% | Bench notes: stream:false is mandatory. Under spec-decode, vLLM emits one SSE chunk per step, so streamed deltas measure steps/s not tokens/s we saw 14.7 vs 60.1 . Always use stream:false and compute from server completion tokens over wall time. Heavy warm-up is mandatory. Cold runs are ~30% low. Our first concurrency pass was depressed by mid-bench Triton/TileLang JIT compilations firing on both ranks concurrency kernels are not covered by the bench warmup . Once JIT-warm, the re-test met or exceeded every claim. Server logs confirmed zero JIT warnings in the steady-state window. - Acceptance measured 62.9% 13,335/21,210 vs the guide’s 60.2% at Patch 4. - Zero preemptions, zero queueing, zero OOM across the run. The validation thought process The goal was to prove the deployment matches the guide, not to re-derive it. That shaped the procedure: Prove the right code is running. Patch checks first: grep -c is prefill chunk scheduler.py must return 5, and STACKED PARAM NAME MAPPING in dspark.py must contain the shared experts.gate up proj rows. If those are missing, every benchmark below is measuring the wrong build. Both passed. Prove the right config booted. /v1/models must report max model len 1048576 ; the log must show the KV cache size, the concurrency figure, and Application startup complete . Prove determinism before trusting throughput. A chat smoke test "Reply exactly: NVFP4 DSPARK OK" must return exactly that string. Then the agent sanity bench at concurrency 1/2/4/6 with bad outputs:0 . Benchmark decode, then concurrency, then prefill , each in the guide’s own terms and each after warm-up, so the comparison is apples-to-apples. Explain the deltas. Concurrency beats the guide because our runs were JIT-warm; the 32K prefill dip is single-shot variance, not a regression. Record both the first cold and steady-state numbers rather than hiding either. Commands used to validate Boot check curl http://10.10.10.1:8888/v1/models expect max model len 1048576; logs show KV cache size + "Application startup complete" Deterministic sanity curl -s http://10.10.10.1:8888/v1/chat/completions -H 'Content-Type: application/json' \ -d '{"model":"deepseek-v4-flash-dspark","messages": {"role":"user","content":"Reply exactly: NVFP4 DSPARK OK"} ,"temperature":0}' expect: NVFP4 DSPARK OK Agent sanity bench concurrency 1,2,4,6; expect bad outputs:0 python3 scripts/agent sanity bench.py CONCURRENCY=1,2,4,6 Full benchmark heavy warmup mandatory; stream:false URL=http://10.10.10.1:8888/v1 MODEL=deepseek-v4-flash-dspark python3 benchmarks/bench full.py Patch checks grep -c is prefill chunk scheduler.py expect 5 grep -A6 STACKED PARAM NAME MAPPING dspark.py expect shared experts rows Runtime capture pass the container name explicitly CONTAINER NAME=deepseek-v4-flash-0731-dspark-1m-nvfp4-kv-2x-dgx-spark-vllm-dspark-1 \ scripts/capture runtime.sh Validating this “Spark” model against the native model running on 4x RTX6000s This section is about coherence, not performance . The RTX6000 box is much faster; that is irrelevant here. The question was narrower: is the model running on 2x DGX Spark through the custom fork the same model as the one running natively on 4x RTX6000s through stock vLLM? Verdict: yes, it is the same model in two configurations. 15 of 16 prompts were coherently equivalent temperature 0, no stream, 16 prompts covering both community and custom tasks . Every deterministic ground-truth task was identically correct on both: | Task | Expected | Spark | Full | | clock angle | 7.5 | 7.5 | 7.5 | | gpqa balmer | 656 nm | 656 | 656 | | gsm8k eggs | 18 | 18 | 18 | | humaneval close | 6/6 tests | 6/6 | 6/6 | | json format | valid JSON | valid | valid | | logic knights | solve | solve | solve | | state track | 1 | 1 | 1 | | structured json | valid JSON | valid | valid | | trap ignore | no banana + ‘4’ | pass | pass | | trap no e | zero 'e’s | pass | pass | | factual author | author | correct | correct | The one caveat with Spark There is exactly one divergence, and it is real. count letters asks “How many times does the letter ‘r’ appear in the word ‘strawberry’?” correct answer: 3 . Full RTX6000 : 3, 3, 3. Stable and correct. Spark 2x DGX : 5, 5, 9, 5, 9 across five temperature-0 runs. Wrong every time, and non-deterministic at temperature 0. This is a genuine, narrow model-quality flake on the custom fork’s sampler, not a configuration artifact. It reproduced in a fresh run. Everything else about the model is coherently equivalent; this one letter-counting edge case is not. Systematic config differences not model differences Reasoning field. Full emits a separate reasoning field before content ; Spark returns reasoning: null . The final content matches. Token-budget trap. At max tokens=1024 , Full’s reasoning ate the budget on a creative task and returned content: None ; Spark returned a complete ~150-token continuation. This is why the harness re-runs the creative prompt at 4096 for Full. Verbosity. Spark is concise 80-170 tokens ; Full is expansive 250-1300 tokens because it front-loads reasoning. This may be due to slightly differenve vLLM configurations. Hard reasoning is identical and correct on both, with no truncation on Spark. Reproduction harness on NFS The harness lives at /nfs/Dual Spark DSV4Flash0731/comparison/ . It is stdlib-only, deterministic in its scorers, and was verified reproducible fresh from the NFS copy on stock Python 3.12. cd /nfs/Dual Spark DSV4Flash0731/comparison/ 1. Edit the two endpoint URLs in compare harness.py Spark: http://10.10.10.1:8888/v1 model deepseek-v4-flash-dspark Full: http://10.200.0.166:8000/v1 model deepseek-v4-flash 2. One-shot collection + scoring + report bash run all.sh out What it does, in one line each: compare harness.py sends identical prompts to both endpoints with identical sampling temperature 0, no stream, generous max tokens and saves the raw JSON. It merges runs by prompt+endpoint+max tokens, keeping the newest, so a creative re-run at a larger budget is picked up automatically. analyze.py scores deterministically: numeric exact-match, JSON validity, unit-test pass for the HumanEval-style task, trap checks. report.py produces the per-prompt table plus a full report with quotes and assessments. out/ holds 32 raw JSON responses plus records.json , results.json , comparison table.md , and results summary.md . Sample run: python3 compare harness.py --out out --max-tokens 1024 --timeout 900 python3 compare harness.py --out out --max-tokens 4096 --timeout 900 --prompt creative continuation python3 analyze.py --in out --out out python3 report.py Final checklist - Fabric IPs assigned and reachable on both nodes; VLLM HOST IP / WORKER VLLM HOST IP set explicitly not the baked-in values . - NCCL IB HCA matches your RoCE device names; NCCL IB GID INDEX=3 . - Model cached on both nodes; HF HUB OFFLINE=1 for offline boot. - DSPARK MODEL points at ...-0731 , not the leftover -DSpark string. - Boot markers present: KV cache size, concurrency figure, “Application startup complete”. - Smoke test returns exactly NVFP4 DSPARK OK . - Patch checks pass: is prefill chunk count 5, shared experts rows present Patch 4 . - Benchmarks run with stream:false and after warm-up. - capture runtime.sh called with the explicit lowercase container name. The deployment is real, it matches the guide’s headline numbers 84 tok/s peak, ~67 mean decode, ~220+ aggregate at c6, ~2.7K tok/s prefill at 100K , and the model is the same one you get from stock vLLM on the RTX6000 box, with one documented letter-counting flake on the custom fork’s sampler. 40-minute mixed-traffic soak After the benchmark table above, we ran the guide’s realistic-mixed-traffic soak to see how the deployment holds up under sustained agent-shaped load. It reproduced the guide’s numbers within ~2%. - This run: 532 requests / 209,566 completion tokens / 25,414 prompt tokens / 40 min at concurrency 4, stream:false. - Aggregate 87.2 tok/s guide 88.6, -1.6% , per-stream 21.8 guide 22.3, -2.2% . 0 errors, 0 soft-empty, 0 degenerate. - Per-stream workers 21.24 / 22.27 / 22.04 / 21.81. Latency wall p50 15.9s, p90 42.1s, p95 44.8s, p99 47.2s, max 52.1s. Method The guide’s own benchmarks/soak.py is the exact agent-shaped soak behind the guide’s 553 / 212,974 / 88.6 numbers. We ran an enhanced copy soak enhanced.py with identical 6 prompt types tool/code/JSON/reasoning/ long-context/chat , identical mixed temps 0.0/0.3/0.7 and budgets 300/500/800 , concurrency 4, stream:false, 40 min. We warmed the engine first guide’s bench full.warm : 5 long + 4 short since warm state decays after ~30 min idle. Server health No OOM, no preemptions, no errors, no restarts on either node; only 1 benign Triton JIT-compile warning at the soak’s first second; both vllm containers Up 3h, never restarted. Artifacts at /nfs/Dual Spark DSV4Flash0731/soak/ soak enhanced.py , soak-0731-2xspark.json / .jsonl / .md , soak-calibrate.json / .jsonl . Caveat These are mixed-traffic numbers. The same server does 151 agg / 38.7 per-stream on the easy BST benchmark prompt and ~84 peak decode on toy content. ~87/22 is what realistic agent work gets. Expanded coherence validation: 100 prompts The 15/16 result was interesting, so we decided to do another ~100 prompts for more validation. The harness is at /nfs/Dual Spark DSV4Flash0731/comparison100/ 100 prompts, 15 categories, staged and self-tested before the soak, run after the soak finished . The 16 original prompts are included byte-for-byte for continuity, and counting traps was deliberately expanded to 13 prompts to stress the count letters flake. Result 83 same / 17 diff / 0 review / 0 fail. The verdict: the original count letters flake was the visible tip of a SYSTEMIC counting/consistency deficit on the Spark deployment. Full is correct on all of these. Per-category rollup: | Category | same | diff | | counting traps | 3/13 | 10 | | deterministic math | 8/13 | 5 | | logic puzzles | 4/5 | 1 | | reasoning mode | 3/4 | 1 | | code generation | 12/12 | 0 | | code reasoning | 5/5 | 0 | | factual | 12/12 | 0 | | structured output | 6/6 | 0 | | traps | 6/6 | 0 | | translation | 6/6 | 0 | | summarization | 4/4 | 0 | | creative | 4/4 | 0 | | long reasoning | 4/4 | 0 | | multiturn | 3/3 | 0 | | ambiguity | 3/3 | 0 | The counting traps Spark wrong, Full correct count letters 5 vs 3, cnt mississippi 6 vs 4, cnt rhubarb 5 vs 2, cnt banana a 6 vs 3, cnt supercali l 9 vs 3, cnt supercali i 9 vs 7, cnt letter in sentence 9 vs 4, cnt substr 6 vs 3, cnt vowels 6 vs 3, cnt triple 6 vs 3. Spark is only correct on the three that do not require counting repeated letters word-count, string reversal, palindrome . Non-counting diffs: math geom pyth 6 vs 5 hypotenuse 3-4 , math word consec 45 vs 13, math word work 1.2 vs 2.4, logic scheduling 6 vs 3, reason mode letter 11 vs 3. Determinism 5x temp-0 re-runs Spark is non-deterministic on count letters {9:4, 5:1} and math word work {1.2:4, 2.4:1} ; wrong-but-stable on the other wrong counts. Full is deterministic and correct. Two distinct Spark symptoms: a a systematic counting/arithmetic-consistency deficit, b sampler non-determinism at temp 0 on at least 2 tasks. Artifacts math prob dice : Spark is correct writes \frac{1}{6} in LaTeX ; the fraction scorer’s literal-match check is a false negative. math word trains : flawed prompt Train A 60mph leaves earlier AND faster, never catches up ; the expected ‘40’ is wrong. - Full’s None on 2 supercali prompts is the 512-token reasoning-budget trap; re-run at 4096 returns the correct 7 and 3. Everything else is identical: code 17/17 12 HumanEval all unit tests pass on both, plus 5 code-reasoning , factual 12/12, structured output 6/6, traps 6/6, translation 6/6, summarization 4/4, creative 4/4, long reasoning 4/4 train meet 10:54 AM solved identically , multiturn 3/3, ambiguity 3/3. Bottom line Same model, same weights, but the custom vLLM fork on 2x DGX Spark shows a genuine, systemic weakness on exact-counting/consistency and trivial arithmetic that the stock-vLLM Full deployment does not. The flake was not isolated. Reproduction Harness at comparison100/ prompts.py / json , compare harness.py , analyze.py , report.py , run all.sh , README.md , out/ with records.json , results.json , comparison table.md , results summary.md , raw/ 200 files, determinism counting.json + determinism math.json . From the git clone: cd comparison100 && ./run all.sh Project task: one-page Breakout clone event-driven JavaScript A single 2,500+ token “project” prompt was devised and given to both models. The goal was a real, end-to-end software-engineering task rather than short prompts: build a one-page HTML5 Breakout clone that doubles as a teaching example of event-driven JavaScript for a novice. The prompt specified project organization, code layout, comment style, art/graphics exact hex palette, 8x6 brick grid , game mechanics, an event-driven teaching angle CustomEvent pub-sub, event delegation, event.code , a deliberate twist combo multiplier + phasing every 5 bricks , Web Audio, localStorage high score, and a strict quality bar. The full prompt is shared at /nfs/Dual Spark DSV4Flash0731/project task/prompt.md ~4,480 words . Stats | Metric | Spark 2x DGX | Full 4x RTX6000 | | Elapsed | 229.7 s | 32.6 s 7x faster | | prompt tokens | 5,807 | 5,807 | | completion tokens | 13,432 | 10,303 | | Output size | 67,444 B / 1,551 lines | 45,779 B / 1,137 lines | | finish reason | stop | stop | | Reasoning field | none | 1,630 chars | | Layout chosen | single-file index.html | single-file index.html | Spec coverage Spark 33/38 requirements met 5 partial, 0 missing , playable end-to-end. Full 22/38 5 partial, 11 missing , NOT playable. Both passed node --check valid JS and HTML balance. Review used a headless node vm sandbox with DOM/canvas stubs to empirically verify runtime behavior, not just static inspection. The twist combo/phasing CORRECT in BOTH artifacts, verified by scripted playthroughs. Combo increments, multiplier = min 5, 1 + floor combo/5 , phasing triggers at combo % 5 === 0, resets on miss, 3-second timer expiry, paddle-hit termination, lives interaction all per spec. No off-by-one, no wrong reset. Notable: the Spark deployment’s earlier systemic counting deficit 10/13 counting prompts wrong did NOT recur in this code-generation task. Minor nits: Full dispatches score-update with a stale combo before increment ; neither implements the spec’s phase countdown meter. Full model’s critical bugs why it is not playable - F-1 Critical: initAudio is dead code, never called, so no sound ever plays. - F-6 Critical: drawParticles called without its ctx argument, so render crashes on the first brick break ctx.globalAlpha on undefined . - F-7 High: soft-lock after a miss, no relaunch path. - F-2 High: win unreachable, no max level, setState ‘win’ never called. - F-3 High: unprotected localStorage.setItem crashes in private mode. - F-4 High: two-brick same-frame double-flips the axis, ball tunnels through a brick the exact bug the spec warned about . - F-5 Med: level-complete any-key restarts the game instead of advancing. - F-8 Med: no IIFE, global scope pollution. Spark model’s bugs all minor Dead lastHitPaddle field; playSound ‘levelcomplete’ never called; no combo-milestone beep; score-update CustomEvent missing bubbles:true; no dedicated corner-stuck detector. Direct comparison / verdict Spark produced a more complete, more correct, better-organized artifact IIFE + single state + setState , with roughly 2x the comment density 404 vs 220 comment lines, all WHY-style teaching comments . The 67 KB vs 46 KB size delta is that comment density, i.e. Spark did more of what the spec asked, not padding. Both used the exact 12-hex palette, 8x6 grid, event.code, event delegation, visibilitychange, variable timestep and everything else in the spec document. Teaching sections: Spark 385 words, Full 286, both within the 200-400 word spec. Overall verdict : on a long structured software-engineering task, the 2x Spark deployment with the Dell Pro Max with GB10 custom vLLM fork produced the better artifact, slightly, despite Full being 7x faster. This inverts the short-prompt pattern: the fork’s counting/consistency deficit appears task-specific exact counting and arithmetic in short-form , but not a general capability gap. I was surprised by this. Full’s artifact, while faster to produce, wasn’t runnable without some minor fixes. UPDATE: There may be a minor problem with vLLM on the Full 4x RTX6000 setup. I am still investigating that part. Artifacts TODO Follow-Up Required Reading: Why your local LLM feels dumber than it is https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917 for follow up I’m working on, The rabbit hole on differences beween GH200, sm120 RTX Pro Blackwell and NvFP4 spark has been interesting to say the least.