# Dual Sparks in nvfp4 vs 4x RTX Pro 6000 with native DeepSeek V4 0731 -- Quants and Speed > Source: > Published: 2026-08-22 00:21:03+00:00 DeepSeek-V4-Flash-0731 on 2x DGX Spark: 1M Context, NVFP4 KV Cache A practical, field-tested companion to the upstream guide: [GitHub - tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark: DeepSeek V4 Flash DSpark 1M NVFP4 KV recipe for 2x DGX Spark · GitHub](https://github.com/tonyd2wild/DeepSeek-V4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark) This document is written for a homelab enthusiast who already owns two DGX Spark units (GB10, SM120), has a ConnectX/QSFP fabric between them, runs Docker, and wants a single 304B-parameter model serving at 1M context with an NVFP4 KV cache. It is a companion, not a replacement. Read the guide first, then use this to avoid the footguns we hit on a real deployment. Everything below was verified on our own pair of Sparks, not copied from the README. [!IMPORTANT] This recipe runs a **custom vLLM fork**, not stock vLLM. Do not point stock > vLLM (main, 0.24, or 0.25.2) at this model on DGX Spark. See > [Why not stock vLLM](#why-not-stock-vllm) below. The shape of the thing | Component | Value | | Hardware | 2x DGX Spark (GB10, SM120), 1 GPU per node | | Tensor parallelism | TP=2 (one GPU per node) | | Fabric | Dual ConnectX-7 200 Gb/s RoCE (RoCEv2, GID index 3) | | Base image | `ghcr.io/bjk110/vllm-spark:unholy-fusion-prod-ready` (public, no auth, ~21.2 GiB) | | Runtime | vLLM 0.21.1rc1.dev339+g1967a5627bc3, FlashInfer 0.6.12, TileLang 0.1.9, Python 3.12 at `/opt/env` | | Model | `deepseek-ai/DeepSeek-V4-Flash-0731` , rev `7872f01b` , 166.9 GB, 48 safetensors shards | | KV cache dtype | `nvfp4_ds_mla` (the “NVFP4” in the repo name is the KV cache, not the weights) | | Context | 1,048,576 tokens (YaRN factor 16 over 65,536; 1M is the calibrated ceiling) | | API | `http://:8888/v1` | The build produces four images per node (overlay, stage-a, stage-b, stage-c); **stage-c is canonical** and is what you run. Key `.env.dspark` values (from our working config) ``` WORKER_HOST= MASTER_ADDR=10.10.10.1 MASTER_PORT=25440 VLLM_HOST_IP=10.10.10.1 WORKER_VLLM_HOST_IP=10.10.10.2 NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1 NCCL_SOCKET_IFNAME=enp1s0f1np1 NCCL_IB_GID_INDEX=3 HF_CACHE=/home/spark2/.cache/huggingface DSPARK_MODEL=deepseek-ai/DeepSeek-V4-Flash-0731 ``` Our fabric used `10.10.10.1/.2` ; the guide uses `192.168.192.0/24` . Your subnet and device names will differ. The values above are the ones that worked for us. Default config (current best per the guide, and what we ran): ``` MAX_MODEL_LEN=1048576 MAX_NUM_SEQS=6 GPU_MEMORY_UTILIZATION=0.78 MTP_NUM_TOKENS=5 kv-cache-dtype nvfp4_ds_mla speculative config {"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"} --distributed-executor-backend mp ``` Deviating from `MAX_NUM_SEQS=6` , `GPU_MEMORY_UTILIZATION=0.78` , or k=5 reproduces documented crashes. The model card recommends `num_speculative_tokens=7` , but this drafter’s `dspark_block_size` is 5 and k=7 does not work on it. Use 5. Launch sequence ``` ./build-dspark-vllm-runtime.sh # build overlay + stage A/B/C on head AND worker ./prepare-dspark-model-cache.sh # snapshot_download + verify + mirror to worker ./start-deepseek-v4-flash-dspark.sh # worker-first launch + smoke test ``` The API lands at `http://:8888/v1` . Boot markers to look for: ``` GPU KV cache size 1,492,347 tokens Maximum concurrency for 1,048,576 tokens per request: 1.42x Application startup complete. ``` Then the test: `chat` with `"Reply exactly: NVFP4 DSPARK OK"` should return exactly that string. Where the guide needs adjustment (we hit all of these) The guide is good, but it assumes a specific lab. On your hardware and setup, you will need to fix these. 1. Baked-in `VLLM_HOST_IP` footgun (fatal if ignored) The base image has `VLLM_HOST_IP=192.168.192.1` baked in. On any node whose fabric IP is not `192.168.192.1` , the worker dies immediately with: ``` zmq.error.ZMQError: Cannot assign requested address ``` Our image also carried a stale `NODE_IP=192.168.50.58` . Fix: set `VLLM_HOST_IP` (head) and `WORKER_VLLM_HOST_IP` (worker) explicitly per node, and override the stale `NODE_IP` . Do not rely on anything baked into the image. 2. `NCCL_IB_HCA` is per-node, not uniform The image bakes `NCCL_IB_HCA=rocep1s0f0` , but RoCE device names are not uniform across units. Auto-detect the Up RoCE device whose netdev carries the fabric IP. On both of our nodes that resolved to: ``` NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1 NCCL_IB_GID_INDEX=3 # RoCEv2 ``` Verify with `ibstat` /`ibv_devinfo` rather than trusting the baked value. 3. The model must be on BOTH nodes (TP=2 gotcha) With TP=2 every node needs the full 166.9 GB of weights. Download once (to NFS or the head), then rsync to both nodes’ local HF caches. With `HF_HUB_OFFLINE=1` the runtime resolves the model fully offline, which is how you avoid a surprise 167 GB re-download at boot. 4. Leftover `DSPARK_MODEL` string `.env.dspark.example` defaults `DSPARK_MODEL` to `deepseek-ai/DeepSeek-V4-Flash-DSpark` , which is a leftover name. The real HF repo is `deepseek-ai/DeepSeek-V4-Flash-0731` . Point it at the 0731 local cache path and run with `HF_HUB_OFFLINE=1` . 5. `capture_runtime.sh` container-name mismatch The script’s default `CONTAINER_NAME` is the title-cased basename of the directory, which does not match the lowercase deploy container name. If you do not pass it explicitly, the captured files come back empty. Pass: ``` CONTAINER_NAME=deepseek-v4-flash-0731-dspark-1m-nvfp4-kv-2x-dgx-spark-vllm-dspark-1 ``` 6. Reasoning-mode gotcha on the full endpoint This only matters if you also run the 4x RTX6000 comparison (see below), but it is worth knowing: the “full” stock-vLLM deployment runs in REASONING mode. It emits a separate `reasoning` field before `content` , and small `max_tokens` budgets get eaten by reasoning. At `max_tokens=16` you get `content: null` ; at 64 you get `"I am DeepSe"` . Use `max_tokens >= 1024` , or hit the raw `/v1/completions` path, which skips reasoning entirely. 7. First boot is slow. Do not panic. First boot takes on the order of 281 s: model load (~79.51 GiB), DeepGEMM warmup, FlashInfer SM120 sparse-MLA autotune, and `torch.compile` . Later boots reuse caches and are much faster. And before you trust any benchmark number, send real traffic first: cold runs are ~30% low until the JIT kernels warm up. 8. Repo rename The repo was renamed from `DeepSeek-v4-Flash-...` to `DeepSeek-v4-Flash-...` (a case change). The old `-v4-` URL still works, but use the current one above. Why not stock vLLM This is the single most important thing to understand. **Stock vLLM cannot serve this model on DGX Spark.** The guide documents this in `UPSTREAM_V024_STATUS.md` : a 2026-07-01 test of vLLM main (which already contained the merged DSpark PR) imported the DSpark modules but did not boot the checkpoint on 2x DGX Spark/SM120. The failures were backend-level: Unknown SF transformation, a Marlin FP4 repack PTX error, the FlashInfer backend rejecting the device, and MXFP4 method incompatibility. A direct bake-off in `RUNTIME-BAKEOFF-2026-07-29.md` against vLLM 0.25.2 (`ghcr.io/anemll/dspark-vllm-gx10:0.1.1` ) confirmed the gap: stock lost 9% on peak, 8% on mean, and 29% at concurrency c6. The gap is step time from the B12X MoE kernels plus a working `torch.compile` path, both of which only exist on the old fork. The out-of-tree patch inventory (one line each) | Patch | What it does | B12X MoE kernels (`b12x_moe.py` , `VLLM_USE_B12X_MOE` ) | The fused MoE kernels that make decode fast on SM120 | B12X MLA sparse attention (`b12x_mla_sparse.py` ) | Sparse-MLA attention for the V4 architecture | DeepSeekV4 model support (`models/deepseek_v4/` ) | Model implementation for SM120/GB10 | Spec-decode proposer Patches 1/2/2b (`dspark_proposer.py` , `dspark.py` , `gpu_model_runner.py` ) | The MTP drafter and its plumbing | Scheduler Patch 3 (`scheduler.py` ) | No spec placeholders on prefill chunks | NVFP4 envelope Stage A/B/C (`nvfp4_ds_mla` dtype) | 416-byte probe validated to a 584-byte padded envelope | Patch 4 (0731, `_STACKED_PARAM_NAME_MAPPING` ) | Adds `shared_experts.gate_up_proj` rows; fixes 12 dropped tensors | `keys-concurrency.patch` | External vendored patch | `official-main-b12x-nvfp4-python.patch` | Experimental, not the default | Patch 4 is the big one for 0731. It fixes 12 dropped tensors and takes mean decode from 32.7 to 55.4 tok/s, peak from 42.0 to 66.1, and acceptance from 25.7% to 60.2%. If your acceptance is stuck in the 20-30% range, Patch 4 is missing. Where upstream is heading The upstream vLLM PRs that this fork tracks or parallels: `vllm-project/vllm#46995` Spec Decode DSpark (merged 2026-07-01) `vllm-project/vllm#41834` SM12x support for DeepSeek V4 Flash (open) `vllm-project/vllm#43477` Enable DeepSeek V4 and GLM-5.1 on SM120 (merged) `vllm-project/vllm#40082` FlashInfer B12X MoE and FP4 GEMM for SM120/121 (merged) `vllm-project/vllm#43332` B12X W4A16 (merged) `vllm-project/vllm#47577` Auto-select FLASHINFER_B12X NVFP4 on SM120 (open) `vllm-project/vllm#49133` DSpark draft under its own model/quant config (open; same failure family as Patch 4) `vllm-project/vllm#49617` Fix speculators dspark attribute loading (open) `vllm-project/vllm#48804` Warm spec-decode Triton kernels at startup (open) `vllm-project/vllm#38476` / `#47629` TRITON_MLA_SPARSE backend (open) `vllm-project/vllm#50576` SM8x support (open issue) `vllm-project/vllm#50660` “0731 cannot be stabilized” (open issue) Repo issues/PRs on the guide itself (renamed from `-v4-` to `-v4-` ; the old URL still works): #1 perf PR, #2 long-context CUDA assert, #3 garble-fix root cause, #4 long-context crash fix PR, #5 max-cudagraph truncation PR, #6 open soft-failure empty content, #7 proposer fixes via image PR, #8 open engine death CUBLAS/NCCL, #11 multi-node 500 on first request, #12 open “where is the docker image”, #13 open portability PR, #14 open sparkrun recipe PR. The takeaway: this is a moving target. The custom fork is the only thing that works today, upstream is converging (several PRs merged), and the open issues above are the ones to watch before you decide to migrate. Model card facts worth knowing `deepseek-ai/DeepSeek-V4-Flash-0731` : 304B params, MIT license, text-generation. `max_position_embeddings` is 1,048,576 with YaRN factor 16 over 65,536 (1M is the calibrated ceiling). Quantization: `expert_dtype fp4` , weights fp8 e4m3 ue8m0 block 128x128. Config: `num_key_value_heads 1` , `head_dim 512` , `q_lora_rank` /`o_lora_rank 1024` , `sliding_window 128` , `index_topk 512` , `num_hash_layers 3` . DSpark is attached; the card recommends `num_speculative_tokens 7` , but the config has `dspark_block_size 5` and the guide documents that k=7 does not work on this drafter (use 5). Benchmark table (measured vs guide claims, steady state) All numbers are tok/s (decode and concurrency are aggregate; prefill is TTFT for 1 output token). Measured with `stream:false` : under spec-decode, streamed deltas measure steps/s, not tokens/s. | Metric | Guide claim | Measured (steady-state) | Delta | | Decode peak | 84.3 | 84.1 | -0.2% | | Decode mean | 67.6 | 67.5 | -0.1% | | Concurrency c1 | 61.0 | 68.8-69.9 | +13-15% | | Concurrency c2 | 91.7 | 104.6-110.7 | +14-21% | | Concurrency c4 | 151.1 | 163.4-169.0 | +8-12% | | Concurrency c6 | 197.3 | 219.2-226.9 | +11-15% | | Prefill 8K | 1,513 | 1,620 | +7.1% | | Prefill 32K | 2,284 | 2,008 | -12.1% (single-shot variance) | | Prefill 100K | 2,639 | 2,690 | +1.9% | Bench notes: `stream:false` is mandatory. Under spec-decode, vLLM emits one SSE chunk per step, so streamed deltas measure steps/s not tokens/s (we saw 14.7 vs 60.1). Always use `stream:false` and compute from server `completion_tokens` over wall time. **Heavy warm-up is mandatory.** Cold runs are ~30% low. Our first concurrency pass was depressed by mid-bench Triton/TileLang JIT compilations firing on both ranks (concurrency kernels are not covered by the bench warmup). Once JIT-warm, the re-test met or exceeded every claim. Server logs confirmed zero JIT warnings in the steady-state window. - Acceptance measured 62.9% (13,335/21,210) vs the guide’s 60.2% at Patch 4. - Zero preemptions, zero queueing, zero OOM across the run. The validation thought process The goal was to prove the deployment matches the guide, not to re-derive it. That shaped the procedure: **Prove the right code is running.** Patch checks first: `grep -c is_prefill_chunk scheduler.py` must return 5, and `_STACKED_PARAM_NAME_MAPPING` in `dspark.py` must contain the `shared_experts.gate_up_proj` rows. If those are missing, every benchmark below is measuring the wrong build. Both passed. **Prove the right config booted.** `/v1/models` must report `max_model_len 1048576` ; the log must show the KV cache size, the concurrency figure, and `Application startup complete` . **Prove determinism before trusting throughput.** A chat smoke test (`"Reply exactly: NVFP4 DSPARK OK"` ) must return exactly that string. Then the agent sanity bench at concurrency 1/2/4/6 with `bad_outputs:0` . **Benchmark decode, then concurrency, then prefill**, each in the guide’s own terms and each after warm-up, so the comparison is apples-to-apples. **Explain the deltas.** Concurrency beats the guide because our runs were JIT-warm; the 32K prefill dip is single-shot variance, not a regression. Record both the first (cold) and steady-state numbers rather than hiding either. Commands used to validate ``` # Boot check curl http://10.10.10.1:8888/v1/models # expect max_model_len 1048576; logs show KV cache size + "Application startup complete" # Deterministic sanity curl -s http://10.10.10.1:8888/v1/chat/completions -H 'Content-Type: application/json' \ -d '{"model":"deepseek-v4-flash-dspark","messages":[{"role":"user","content":"Reply exactly: NVFP4 DSPARK OK"}],"temperature":0}' # expect: NVFP4 DSPARK OK # Agent sanity bench (concurrency 1,2,4,6; expect bad_outputs:0) python3 scripts/agent_sanity_bench.py # CONCURRENCY=1,2,4,6 # Full benchmark (heavy warmup mandatory; stream:false) URL=http://10.10.10.1:8888/v1 MODEL=deepseek-v4-flash-dspark python3 benchmarks/bench_full.py # Patch checks grep -c is_prefill_chunk scheduler.py # expect 5 grep -A6 _STACKED_PARAM_NAME_MAPPING dspark.py # expect shared_experts rows # Runtime capture (pass the container name explicitly!) CONTAINER_NAME=deepseek-v4-flash-0731-dspark-1m-nvfp4-kv-2x-dgx-spark-vllm-dspark-1 \ scripts/capture_runtime.sh ``` Validating this “Spark” model against the native model running on 4x RTX6000s This section is about **coherence, not performance**. The RTX6000 box is much faster; that is irrelevant here. The question was narrower: is the model running on 2x DGX Spark through the custom fork the *same model* as the one running natively on 4x RTX6000s through stock vLLM? **Verdict: yes, it is the same model in two configurations.** 15 of 16 prompts were coherently equivalent (temperature 0, no stream, 16 prompts covering both community and custom tasks). Every deterministic ground-truth task was identically correct on both: | Task | Expected | Spark | Full | | clock_angle | 7.5 | 7.5 | 7.5 | | gpqa_balmer | 656 nm | 656 | 656 | | gsm8k_eggs | 18 | 18 | 18 | | humaneval_close | 6/6 tests | 6/6 | 6/6 | | json_format | valid JSON | valid | valid | | logic_knights | solve | solve | solve | | state_track | 1 | 1 | 1 | | structured_json | valid JSON | valid | valid | | trap_ignore | no banana + ‘4’ | pass | pass | | trap_no_e | zero 'e’s | pass | pass | | factual_author | author | correct | correct | The one caveat with Spark There is exactly one divergence, and it is real. `count_letters` asks “How many times does the letter ‘r’ appear in the word ‘strawberry’?” (correct answer: 3). **Full (RTX6000):** 3, 3, 3. Stable and correct. **Spark (2x DGX):** 5, 5, 9, 5, 9 across five temperature-0 runs. Wrong every time, and non-deterministic at temperature 0. This is a genuine, narrow model-quality flake on the custom fork’s sampler, not a configuration artifact. It reproduced in a fresh run. Everything else about the model is coherently equivalent; this one letter-counting edge case is not. Systematic config differences (not model differences) **Reasoning field.** Full emits a separate `reasoning` field before `content` ; Spark returns `reasoning: null` . The final `content` matches. **Token-budget trap.** At `max_tokens=1024` , Full’s reasoning ate the budget on a creative task and returned `content: None` ; Spark returned a complete ~150-token continuation. This is why the harness re-runs the creative prompt at 4096 for Full. **Verbosity.** Spark is concise (80-170 tokens); Full is expansive (250-1300 tokens) because it front-loads reasoning. This may be due to slightly differenve vLLM configurations. **Hard reasoning** is identical and correct on both, with no truncation on Spark. Reproduction (harness on NFS) The harness lives at `/nfs/Dual_Spark_DSV4Flash0731/comparison/` . It is stdlib-only, deterministic in its scorers, and was verified reproducible fresh from the NFS copy on stock Python 3.12. ``` cd /nfs/Dual_Spark_DSV4Flash0731/comparison/ # 1. Edit the two endpoint URLs in compare_harness.py # Spark: http://10.10.10.1:8888/v1 model deepseek-v4-flash-dspark # Full: http://10.200.0.166:8000/v1 model deepseek-v4-flash # 2. One-shot collection + scoring + report bash run_all.sh out ``` What it does, in one line each: `compare_harness.py` sends identical prompts to both endpoints with identical sampling (temperature 0, no stream, generous `max_tokens` ) and saves the raw JSON. It merges runs by prompt+endpoint+max_tokens, keeping the newest, so a creative re-run at a larger budget is picked up automatically. `analyze.py` scores deterministically: numeric exact-match, JSON validity, unit-test pass for the HumanEval-style task, trap checks. `report.py` produces the per-prompt table plus a full report with quotes and assessments. `out/` holds 32 raw JSON responses plus `records.json` , `results.json` , `comparison_table.md` , and `results_summary.md` . Sample run: ``` python3 compare_harness.py --out out --max-tokens 1024 --timeout 900 python3 compare_harness.py --out out --max-tokens 4096 --timeout 900 --prompt creative_continuation python3 analyze.py --in out --out out python3 report.py ``` Final checklist - Fabric IPs assigned and reachable on both nodes; `VLLM_HOST_IP` / `WORKER_VLLM_HOST_IP` set explicitly (not the baked-in values). - `NCCL_IB_HCA` matches your RoCE device names; `NCCL_IB_GID_INDEX=3` . - Model cached on **both** nodes; `HF_HUB_OFFLINE=1` for offline boot. - `DSPARK_MODEL` points at `...-0731` , not the leftover `-DSpark` string. - Boot markers present: KV cache size, concurrency figure, “Application startup complete”. - Smoke test returns exactly `NVFP4 DSPARK OK` . - Patch checks pass: `is_prefill_chunk` count 5, `shared_experts` rows present (Patch 4). - Benchmarks run with `stream:false` and after warm-up. - `capture_runtime.sh` called with the explicit lowercase container name. The deployment is real, it matches the guide’s headline numbers (84 tok/s peak, ~67 mean decode, ~220+ aggregate at c6, ~2.7K tok/s prefill at 100K), and the model is the same one you get from stock vLLM on the RTX6000 box, with one documented letter-counting flake on the custom fork’s sampler. 40-minute mixed-traffic soak After the benchmark table above, we ran the guide’s realistic-mixed-traffic soak to see how the deployment holds up under sustained agent-shaped load. It reproduced the guide’s numbers within ~2%. - This run: 532 requests / 209,566 completion tokens / 25,414 prompt tokens / 40 min at concurrency 4, stream:false. - Aggregate **87.2 tok/s** (guide 88.6, -1.6%), per-stream **21.8** (guide 22.3, -2.2%). 0 errors, 0 soft-empty, 0 degenerate. - Per-stream workers 21.24 / 22.27 / 22.04 / 21.81. Latency wall p50 15.9s, p90 42.1s, p95 44.8s, p99 47.2s, max 52.1s. Method The guide’s own `benchmarks/soak.py` is the exact agent-shaped soak behind the guide’s 553 / 212,974 / 88.6 numbers. We ran an enhanced copy (`soak_enhanced.py` ) with identical 6 prompt types (tool/code/JSON/reasoning/ long-context/chat), identical mixed temps (0.0/0.3/0.7) and budgets (300/500/800), concurrency 4, stream:false, 40 min. We warmed the engine first (guide’s `bench_full.warm` : 5 long + 4 short) since warm state decays after ~30 min idle. Server health No OOM, no preemptions, no errors, no restarts on either node; only 1 benign Triton JIT-compile warning at the soak’s first second; both vllm containers Up 3h, never restarted. Artifacts at `/nfs/Dual_Spark_DSV4Flash0731/soak/` (`soak_enhanced.py` , `soak-0731-2xspark.json` /`.jsonl` /`.md` , `soak-calibrate.json` /`.jsonl` ). Caveat These are mixed-traffic numbers. The same server does 151 agg / 38.7 per-stream on the easy BST benchmark prompt and ~84 peak decode on toy content. ~87/22 is what realistic agent work gets. Expanded coherence validation: 100 prompts The 15/16 result was interesting, so we decided to do another ~100 prompts for more validation. The harness is at `/nfs/Dual_Spark_DSV4Flash0731/comparison100/` (100 prompts, 15 categories, staged and self-tested before the soak, run after the soak finished). The 16 original prompts are included byte-for-byte for continuity, and `counting_traps` was deliberately expanded to 13 prompts to stress the `count_letters` flake. Result **83 same / 17 diff / 0 review / 0 fail.** The verdict: the original `count_letters` flake was the visible tip of a **SYSTEMIC counting/consistency deficit** on the Spark deployment. Full is correct on all of these. Per-category rollup: | Category | same | diff | | counting_traps | 3/13 | 10 | | deterministic_math | 8/13 | 5 | | logic_puzzles | 4/5 | 1 | | reasoning_mode | 3/4 | 1 | | code_generation | 12/12 | 0 | | code_reasoning | 5/5 | 0 | | factual | 12/12 | 0 | | structured_output | 6/6 | 0 | | traps | 6/6 | 0 | | translation | 6/6 | 0 | | summarization | 4/4 | 0 | | creative | 4/4 | 0 | | long_reasoning | 4/4 | 0 | | multiturn | 3/3 | 0 | | ambiguity | 3/3 | 0 | The counting traps (Spark wrong, Full correct) `count_letters` 5 vs 3, `cnt_mississippi` 6 vs 4, `cnt_rhubarb` 5 vs 2, `cnt_banana_a` 6 vs 3, `cnt_supercali_l` 9 vs 3, `cnt_supercali_i` 9 vs 7, `cnt_letter_in_sentence` 9 vs 4, `cnt_substr` 6 vs 3, `cnt_vowels` 6 vs 3, `cnt_triple` 6 vs 3. Spark is only correct on the three that do not require counting repeated letters (word-count, string reversal, palindrome). Non-counting diffs: `math_geom_pyth` 6 vs 5 (hypotenuse 3-4), `math_word_consec` 45 vs 13, `math_word_work` 1.2 vs 2.4, `logic_scheduling` 6 vs 3, `reason_mode_letter` 11 vs 3. Determinism (5x temp-0 re-runs) Spark is non-deterministic on `count_letters` ({9:4, 5:1}) and `math_word_work` ({1.2:4, 2.4:1}); wrong-but-stable on the other wrong counts. Full is deterministic and correct. Two distinct Spark symptoms: (a) a systematic counting/arithmetic-consistency deficit, (b) sampler non-determinism at temp 0 on at least 2 tasks. Artifacts `math_prob_dice` : Spark is correct (writes `\frac{1}{6}` in LaTeX); the fraction scorer’s literal-match check is a false negative. `math_word_trains` : flawed prompt (Train A 60mph leaves earlier AND faster, never catches up); the expected ‘40’ is wrong. - Full’s `None` on 2 supercali prompts is the 512-token reasoning-budget trap; re-run at 4096 returns the correct 7 and 3. Everything else is identical: code 17/17 (12 HumanEval all unit tests pass on both, plus 5 code-reasoning), factual 12/12, structured output 6/6, traps 6/6, translation 6/6, summarization 4/4, creative 4/4, long reasoning 4/4 (`train_meet` 10:54 AM solved identically), multiturn 3/3, ambiguity 3/3. Bottom line Same model, same weights, but the custom vLLM fork on 2x DGX Spark shows a genuine, systemic weakness on exact-counting/consistency and trivial arithmetic that the stock-vLLM Full deployment does not. The flake was not isolated. Reproduction Harness at `comparison100/` (`prompts.py` /`json` , `compare_harness.py` , `analyze.py` , `report.py` , `run_all.sh` , `README.md` , `out/` with `records.json` , `results.json` , `comparison_table.md` , `results_summary.md` , `raw/` 200 files, `determinism_counting.json` + `determinism_math.json` ). From the git clone: ``` cd comparison100 && ./run_all.sh ``` Project task: one-page Breakout clone (event-driven JavaScript) A single 2,500+ token “project” prompt was devised and given to both models. The goal was a real, end-to-end software-engineering task rather than short prompts: build a one-page HTML5 Breakout clone that doubles as a teaching example of event-driven JavaScript for a novice. The prompt specified project organization, code layout, comment style, art/graphics (exact hex palette, 8x6 brick grid), game mechanics, an event-driven teaching angle (CustomEvent pub-sub, event delegation, event.code), a deliberate twist (combo multiplier + phasing every 5 bricks), Web Audio, localStorage high score, and a strict quality bar. The full prompt is shared at `/nfs/Dual_Spark_DSV4Flash0731/project_task/prompt.md` (~4,480 words). Stats | Metric | Spark (2x DGX) | Full (4x RTX6000) | | Elapsed | 229.7 s | 32.6 s (7x faster) | | prompt_tokens | 5,807 | 5,807 | | completion_tokens | 13,432 | 10,303 | | Output size | 67,444 B / 1,551 lines | 45,779 B / 1,137 lines | | finish_reason | stop | stop | | Reasoning field | none | 1,630 chars | | Layout chosen | single-file index.html | single-file index.html | Spec coverage Spark 33/38 requirements met (5 partial, 0 missing), playable end-to-end. Full 22/38 (5 partial, 11 missing), NOT playable. Both passed `node --check` (valid JS) and HTML balance. Review used a headless node vm sandbox with DOM/canvas stubs to empirically verify runtime behavior, not just static inspection. The twist (combo/phasing) CORRECT in BOTH artifacts, verified by scripted playthroughs. Combo increments, multiplier = min(5, 1 + floor(combo/5)), phasing triggers at combo % 5 === 0, resets on miss, 3-second timer expiry, paddle-hit termination, lives interaction all per spec. No off-by-one, no wrong reset. Notable: the Spark deployment’s earlier systemic counting deficit (10/13 counting prompts wrong) did NOT recur in this code-generation task. Minor nits: Full dispatches score-update with a stale combo (before increment); neither implements the spec’s phase countdown meter. Full model’s critical bugs (why it is not playable) - F-1 Critical: initAudio() is dead code, never called, so no sound ever plays. - F-6 Critical: drawParticles() called without its ctx argument, so render() crashes on the first brick break (ctx.globalAlpha on undefined). - F-7 High: soft-lock after a miss, no relaunch path. - F-2 High: win unreachable, no max level, setState(‘win’) never called. - F-3 High: unprotected localStorage.setItem crashes in private mode. - F-4 High: two-brick same-frame double-flips the axis, ball tunnels through a brick (the exact bug the spec warned about). - F-5 Med: level-complete any-key restarts the game instead of advancing. - F-8 Med: no IIFE, global scope pollution. Spark model’s bugs (all minor) Dead lastHitPaddle field; playSound(‘levelcomplete’) never called; no combo-milestone beep; score-update CustomEvent missing bubbles:true; no dedicated corner-stuck detector. Direct comparison / verdict Spark produced a more complete, more correct, better-organized artifact (IIFE + single state + setState), with roughly 2x the comment density (404 vs 220 comment lines, all WHY-style teaching comments). The 67 KB vs 46 KB size delta is that comment density, i.e. Spark did more of what the spec asked, not padding. Both used the exact 12-hex palette, 8x6 grid, event.code, event delegation, visibilitychange, variable timestep and everything else in the spec document. Teaching sections: Spark 385 words, Full 286, both within the 200-400 word spec. **Overall verdict**: on a long structured software-engineering task, the 2x Spark deployment with the Dell Pro Max with GB10 (custom vLLM fork) produced the better artifact, slightly, despite Full being 7x faster. This inverts the short-prompt pattern: the fork’s counting/consistency deficit appears task-specific (exact counting and arithmetic in short-form), but not a general capability gap. I was surprised by this. Full’s artifact, while faster to produce, wasn’t runnable without some minor fixes. UPDATE: There may be a minor problem with vLLM on the Full 4x RTX6000 setup. I am still investigating that part. Artifacts TODO Follow-Up Required Reading: [Why your local LLM feels dumber than it is](https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917) for follow up I’m working on, The rabbit hole on differences beween GH200, sm120 RTX Pro Blackwell and NvFP4 spark has been *interesting* to say the least.