DeepSeek-V4-Flash-0731 on 2x DGX Spark: 1M Context, NVFP4 KV Cache
A practical, field-tested companion to the upstream guide:
This document is written for a homelab enthusiast who already owns two DGX Spark units (GB10, SM120), has a ConnectX/QSFP fabric between them, runs Docker, and wants a single 304B-parameter model serving at 1M context with an NVFP4 KV cache. It is a companion, not a replacement. Read the guide first, then use this to avoid the footguns we hit on a real deployment. Everything below was verified on our own pair of Sparks, not copied from the README.
[!IMPORTANT]
This recipe runs a custom vLLM fork, not stock vLLM. Do not point stock > vLLM (main, 0.24, or 0.25.2) at this model on DGX Spark. See > Why not stock vLLM below.
The shape of the thing
| Component |
Value |
| Hardware |
2x DGX Spark (GB10, SM120), 1 GPU per node |
| Tensor parallelism |
TP=2 (one GPU per node) |
| Fabric |
Dual ConnectX-7 200 Gb/s RoCE (RoCEv2, GID index 3) |
| Base image |
ghcr.io/bjk110/vllm-spark:unholy-fusion-prod-ready (public, no auth, ~21.2 GiB) |
| Runtime |
vLLM 0.21.1rc1.dev339+g1967a5627bc3, FlashInfer 0.6.12, TileLang 0.1.9, Python 3.12 at /opt/env |
| Model |
deepseek-ai/DeepSeek-V4-Flash-0731 , rev 7872f01b , 166.9 GB, 48 safetensors shards |
| KV cache dtype |
nvfp4_ds_mla (the “NVFP4” in the repo name is the KV cache, not the weights) |
| Context |
1,048,576 tokens (YaRN factor 16 over 65,536; 1M is the calibrated ceiling) |
| API |
http://<head-fabric-ip>:8888/v1 |
The build produces four images per node (overlay, stage-a, stage-b, stage-c); stage-c is canonical and is what you run.
Key .env.dspark
values (from our working config)
WORKER_HOST=<worker hostname or IP>
MASTER_ADDR=10.10.10.1
MASTER_PORT=25440
VLLM_HOST_IP=10.10.10.1
WORKER_VLLM_HOST_IP=10.10.10.2
NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1
NCCL_SOCKET_IFNAME=enp1s0f1np1
NCCL_IB_GID_INDEX=3
HF_CACHE=/home/spark2/.cache/huggingface
DSPARK_MODEL=deepseek-ai/DeepSeek-V4-Flash-0731
Our fabric used 10.10.10.1/.2
; the guide uses 192.168.192.0/24
. Your subnet and device names will differ. The values above are the ones that worked for us.
Default config (current best per the guide, and what we ran):
MAX_MODEL_LEN=1048576
MAX_NUM_SEQS=6
GPU_MEMORY_UTILIZATION=0.78
MTP_NUM_TOKENS=5
kv-cache-dtype nvfp4_ds_mla
speculative config {"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"} --distributed-executor-backend mp
Deviating from MAX_NUM_SEQS=6
, GPU_MEMORY_UTILIZATION=0.78
, or k=5
reproduces documented crashes. The model card recommends
num_speculative_tokens=7
, but this drafter’s dspark_block_size
is 5 and k=7 does not work on it. Use 5.
Launch sequence
./build-dspark-vllm-runtime.sh # build overlay + stage A/B/C on head AND worker ./prepare-dspark-model-cache.sh # snapshot_download + verify + mirror to worker ./start-deepseek-v4-flash-dspark.sh # worker-first launch + smoke test
The API lands at http://<head-fabric-ip>:8888/v1
. Boot markers to look for:
GPU KV cache size 1,492,347 tokens Maximum concurrency for 1,048,576 tokens per request: 1.42x Application startup complete.
Then the test: chat
with "Reply exactly: NVFP4 DSPARK OK"
should return exactly that string.
Where the guide needs adjustment (we hit all of these)
The guide is good, but it assumes a specific lab. On your hardware and setup, you will need to fix these.
- Baked-in
VLLM_HOST_IP
footgun (fatal if ignored)
The base image has VLLM_HOST_IP=192.168.192.1
baked in. On any node whose fabric IP is not 192.168.192.1
, the worker dies immediately with:
zmq.error.ZMQError: Cannot assign requested address
Our image also carried a stale NODE_IP=192.168.50.58
. Fix: set
VLLM_HOST_IP
(head) and WORKER_VLLM_HOST_IP
(worker) explicitly per node, and override the stale NODE_IP
. Do not rely on anything baked into the image.
NCCL_IB_HCA
is per-node, not uniform
The image bakes NCCL_IB_HCA=rocep1s0f0
, but RoCE device names are not uniform across units. Auto-detect the Up RoCE device whose netdev carries the fabric IP. On both of our nodes that resolved to:
NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1
NCCL_IB_GID_INDEX=3 # RoCEv2
Verify with ibstat
/ibv_devinfo
rather than trusting the baked value.
- The model must be on BOTH nodes (TP=2 gotcha)
With TP=2 every node needs the full 166.9 GB of weights. Download once (to NFS or the head), then rsync to both nodes’ local HF caches. With HF_HUB_OFFLINE=1
the runtime resolves the model fully offline, which is how you avoid a surprise 167 GB re-download at boot.
- Leftover
DSPARK_MODEL
string
.env.dspark.example
defaults DSPARK_MODEL
to
deepseek-ai/DeepSeek-V4-Flash-DSpark
, which is a leftover name. The real HF repo is deepseek-ai/DeepSeek-V4-Flash-0731
. Point it at the 0731 local cache path and run with HF_HUB_OFFLINE=1
.
capture_runtime.sh
container-name mismatch
The script’s default CONTAINER_NAME
is the title-cased basename of the
directory, which does not match the lowercase deploy container name. If you do
not pass it explicitly, the captured files come back empty. Pass:
CONTAINER_NAME=deepseek-v4-flash-0731-dspark-1m-nvfp4-kv-2x-dgx-spark-vllm-dspark-1
- Reasoning-mode gotcha on the full endpoint
This only matters if you also run the 4x RTX6000 comparison (see below), but it
is worth knowing: the “full” stock-vLLM deployment runs in REASONING mode. It
emits a separate reasoning
field before content
, and small max_tokens
budgets get eaten by reasoning. At max_tokens=16
you get content: null
; at 64 you get "I am DeepSe"
. Use max_tokens >= 1024
, or hit the raw
/v1/completions
path, which skips reasoning entirely.
- First boot is slow. Do not panic.
First boot takes on the order of 281 s: model load (~79.51 GiB), DeepGEMM warmup, FlashInfer SM120 sparse-MLA autotune, and torch.compile
. Later boots reuse caches and are much faster. And before you trust any benchmark number, send real traffic first: cold runs are ~30% low until the JIT kernels warm up.
- Repo rename
The repo was renamed from DeepSeek-v4-Flash-...
to DeepSeek-v4-Flash-...
(a case change). The old -v4-
URL still works, but use the current one above.
Why not stock vLLM
This is the single most important thing to understand. Stock vLLM cannot serve this model on DGX Spark.
The guide documents this in UPSTREAM_V024_STATUS.md
: a 2026-07-01 test of vLLM main (which already contained the merged DSpark PR) imported the DSpark modules but did not boot the checkpoint on 2x DGX Spark/SM120. The failures were backend-level: Unknown SF transformation, a Marlin FP4 repack PTX error, the FlashInfer backend rejecting the device, and MXFP4 method incompatibility.
A direct bake-off in RUNTIME-BAKEOFF-2026-07-29.md
against vLLM 0.25.2 (ghcr.io/anemll/dspark-vllm-gx10:0.1.1
) confirmed the gap: stock lost 9% on peak, 8% on mean, and 29% at concurrency c6. The gap is step time from the B12X MoE kernels plus a working torch.compile
path, both of which only exist on the old fork.
The out-of-tree patch inventory (one line each)
| Patch |
What it does |
B12X MoE kernels (b12x_moe.py , VLLM_USE_B12X_MOE ) |
The fused MoE kernels that make decode fast on SM120 |
B12X MLA sparse attention (b12x_mla_sparse.py ) |
Sparse-MLA attention for the V4 architecture |
DeepSeekV4 model support (models/deepseek_v4/ ) |
Model implementation for SM120/GB10 |
Spec-decode proposer Patches 1/2/2b (dspark_proposer.py , dspark.py , gpu_model_runner.py ) |
The MTP drafter and its plumbing |
Scheduler Patch 3 (scheduler.py ) |
No spec placeholders on prefill chunks |
NVFP4 envelope Stage A/B/C (nvfp4_ds_mla dtype) |
416-byte probe validated to a 584-byte padded envelope |
Patch 4 (0731, _STACKED_PARAM_NAME_MAPPING ) |
Adds shared_experts.gate_up_proj rows; fixes 12 dropped tensors |
keys-concurrency.patch |
External vendored patch |
official-main-b12x-nvfp4-python.patch |
Experimental, not the default |
Patch 4 is the big one for 0731. It fixes 12 dropped tensors and takes mean
decode from 32.7 to 55.4 tok/s, peak from 42.0 to 66.1, and acceptance from
25.7% to 60.2%. If your acceptance is stuck in the 20-30% range, Patch 4 is
missing.
Where upstream is heading
The upstream vLLM PRs that this fork tracks or parallels:
vllm-project/vllm#46995
Spec Decode DSpark (merged 2026-07-01)
vllm-project/vllm#41834
SM12x support for DeepSeek V4 Flash (open)
vllm-project/vllm#43477
Enable DeepSeek V4 and GLM-5.1 on SM120 (merged)
vllm-project/vllm#40082
FlashInfer B12X MoE and FP4 GEMM for SM120/121 (merged)
vllm-project/vllm#43332
B12X W4A16 (merged)
vllm-project/vllm#47577
Auto-select FLASHINFER_B12X NVFP4 on SM120 (open)
vllm-project/vllm#49133
DSpark draft under its own model/quant config (open; same failure family as Patch 4)
vllm-project/vllm#49617
Fix speculators dspark attribute (open)
vllm-project/vllm#48804
Warm spec-decode Triton kernels at startup (open)
vllm-project/vllm#38476
/ #47629
TRITON_MLA_SPARSE backend (open)
vllm-project/vllm#50576
SM8x support (open issue)
vllm-project/vllm#50660
“0731 cannot be stabilized” (open issue)
Repo issues/PRs on the guide itself (renamed from -v4-
to -v4-
; the old URL still works): #1 perf PR, #2 long-context CUDA assert, #3 garble-fix root cause, #4 long-context crash fix PR, #5 max-cudagraph truncation PR, #6 open
soft-failure empty content, #7 proposer fixes via image PR, #8 open engine death CUBLAS/NCCL, #11 multi-node 500 on first request, #12 open “where is the docker image”, #13 open portability PR, #14 open sparkrun recipe PR.
The takeaway: this is a moving target. The custom fork is the only thing that works today, upstream is converging (several PRs merged), and the open issues above are the ones to watch before you decide to migrate.
Model card facts worth knowing
deepseek-ai/DeepSeek-V4-Flash-0731
: 304B params, MIT license,
text-generation. max_position_embeddings
is 1,048,576 with YaRN factor 16 over 65,536 (1M is the calibrated ceiling). Quantization: expert_dtype fp4
, weights fp8 e4m3 ue8m0 block 128x128. Config: num_key_value_heads 1
, head_dim 512
, q_lora_rank
/o_lora_rank 1024
, sliding_window 128
,
index_topk 512
, num_hash_layers 3
. DSpark is attached; the card recommends num_speculative_tokens 7
, but the config has dspark_block_size 5
and the guide documents that k=7 does not work on this drafter (use 5).
Benchmark table (measured vs guide claims, steady state)
All numbers are tok/s (decode and concurrency are aggregate; prefill is TTFT for 1 output token). Measured with stream:false
: under spec-decode, streamed deltas measure steps/s, not tokens/s.
| Metric | Guide claim | Measured (steady-state) | Delta | | Decode peak | 84.3 | 84.1 | -0.2% | | Decode mean | 67.6 | 67.5 | -0.1% | | Concurrency c1 | 61.0 | 68.8-69.9 | +13-15% | | Concurrency c2 | 91.7 | 104.6-110.7 | +14-21% | | Concurrency c4 | 151.1 | 163.4-169.0 | +8-12% | | Concurrency c6 | 197.3 | 219.2-226.9 | +11-15% | | Prefill 8K | 1,513 | 1,620 | +7.1% | | Prefill 32K | 2,284 | 2,008 | -12.1% (single-shot variance) | | Prefill 100K | 2,639 | 2,690 | +1.9% |
Bench notes:
stream:false
is mandatory. Under spec-decode, vLLM emits one SSE chunk per step, so streamed deltas measure steps/s not tokens/s (we saw 14.7 vs 60.1). Always use stream:false
and compute from server completion_tokens
over wall time. Heavy warm-up is mandatory. Cold runs are ~30% low. Our first concurrency pass was depressed by mid-bench Triton/TileLang JIT compilations firing on both ranks (concurrency kernels are not covered by the bench warmup). Once JIT-warm, the re-test met or exceeded every claim. Server logs confirmed zero JIT warnings in the steady-state window.
- Acceptance measured 62.9% (13,335/21,210) vs the guide’s 60.2% at Patch 4.
- Zero preemptions, zero queueing, zero OOM across the run.
The validation thought process
The goal was to prove the deployment matches the guide, not to re-derive it.
That shaped the procedure:
Prove the right code is running. Patch checks first: grep -c is_prefill_chunk scheduler.py
must return 5, and
_STACKED_PARAM_NAME_MAPPING
in dspark.py
must contain the
shared_experts.gate_up_proj
rows. If those are missing, every benchmark
below is measuring the wrong build. Both passed.
Prove the right config booted. /v1/models
must report
max_model_len 1048576
; the log must show the KV cache size, the
concurrency figure, and Application startup complete
. Prove determinism before trusting throughput. A chat smoke test
("Reply exactly: NVFP4 DSPARK OK"
) must return exactly that string. Then
the agent sanity bench at concurrency 1/2/4/6 with bad_outputs:0
. Benchmark decode, then concurrency, then prefill, each in the guide’s
own terms and each after warm-up, so the comparison is apples-to-apples. Explain the deltas. Concurrency beats the guide because our runs were
JIT-warm; the 32K prefill dip is single-shot variance, not a regression.
Record both the first (cold) and steady-state numbers rather than hiding
either.
Commands used to validate
curl http://10.10.10.1:8888/v1/models
curl -s http://10.10.10.1:8888/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"deepseek-v4-flash-dspark","messages":[{"role":"user","content":"Reply exactly: NVFP4 DSPARK OK"}],"temperature":0}'
python3 scripts/agent_sanity_bench.py # CONCURRENCY=1,2,4,6
URL=http://10.10.10.1:8888/v1 MODEL=deepseek-v4-flash-dspark python3 benchmarks/bench_full.py
grep -c is_prefill_chunk scheduler.py # expect 5
grep -A6 _STACKED_PARAM_NAME_MAPPING dspark.py # expect shared_experts rows
CONTAINER_NAME=deepseek-v4-flash-0731-dspark-1m-nvfp4-kv-2x-dgx-spark-vllm-dspark-1 \
scripts/capture_runtime.sh
Validating this “Spark” model against the native model running on 4x RTX6000s
This section is about coherence, not performance. The RTX6000 box is much faster; that is irrelevant here. The question was narrower: is the model running on 2x DGX Spark through the custom fork the same model as the one running natively on 4x RTX6000s through stock vLLM?
Verdict: yes, it is the same model in two configurations. 15 of 16 prompts were coherently equivalent (temperature 0, no stream, 16 prompts covering both community and custom tasks). Every deterministic ground-truth task was identically correct on both:
| Task | Expected | Spark | Full | | clock_angle | 7.5 | 7.5 | 7.5 | | gpqa_balmer | 656 nm | 656 | 656 | | gsm8k_eggs | 18 | 18 | 18 | | humaneval_close | 6/6 tests | 6/6 | 6/6 | | json_format | valid JSON | valid | valid | | logic_knights | solve | solve | solve | | state_track | 1 | 1 | 1 | | structured_json | valid JSON | valid | valid | | trap_ignore | no banana + ‘4’ | pass | pass | | trap_no_e | zero 'e’s | pass | pass | | factual_author | author | correct | correct |
The one caveat with Spark
There is exactly one divergence, and it is real. count_letters
asks “How many times does the letter ‘r’ appear in the word ‘strawberry’?” (correct answer: 3).
Full (RTX6000): 3, 3, 3. Stable and correct. Spark (2x DGX): 5, 5, 9, 5, 9 across five temperature-0 runs. Wrong every time, and non-deterministic at temperature 0.
This is a genuine, narrow model-quality flake on the custom fork’s sampler, not a configuration artifact. It reproduced in a fresh run. Everything else about the model is coherently equivalent; this one letter-counting edge case is not.
Systematic config differences (not model differences)
Reasoning field. Full emits a separate reasoning
field before
content
; Spark returns reasoning: null
. The final content
matches.
Token-budget trap. At max_tokens=1024
, Full’s reasoning ate the budget on a creative task and returned content: None
; Spark returned a complete ~150-token continuation. This is why the harness re-runs the creative prompt at 4096 for Full. Verbosity. Spark is concise (80-170 tokens); Full is expansive
(250-1300 tokens) because it front-loads reasoning. This may be due to slightly differenve vLLM configurations. Hard reasoning is identical and correct on both, with no truncation on Spark.
Reproduction (harness on NFS)
The harness lives at /nfs/Dual_Spark_DSV4Flash0731/comparison/
. It is
stdlib-only, deterministic in its scorers, and was verified reproducible fresh
from the NFS copy on stock Python 3.12.
cd /nfs/Dual_Spark_DSV4Flash0731/comparison/
bash run_all.sh out
What it does, in one line each:
compare_harness.py
sends identical prompts to both endpoints with identical sampling (temperature 0, no stream, generous max_tokens
) and saves the raw JSON. It merges runs by prompt+endpoint+max_tokens, keeping the newest, so a creative re-run at a larger budget is picked up automatically.
analyze.py
scores deterministically: numeric exact-match, JSON validity, unit-test pass for the HumanEval-style task, trap checks.
report.py
produces the per-prompt table plus a full report with quotes and assessments.
out/
holds 32 raw JSON responses plus records.json
, results.json
,
comparison_table.md
, and results_summary.md
.
Sample run:
python3 compare_harness.py --out out --max-tokens 1024 --timeout 900
python3 compare_harness.py --out out --max-tokens 4096 --timeout 900 --prompt creative_continuation
python3 analyze.py --in out --out out
python3 report.py
Final checklist
- Fabric IPs assigned and reachable on both nodes;
VLLM_HOST_IP
/
WORKER_VLLM_HOST_IP
set explicitly (not the baked-in values). #
NCCL_IB_HCA
matches your RoCE device names; NCCL_IB_GID_INDEX=3
.
- Model cached on
both nodes;
HF_HUB_OFFLINE=1
for offline boot. #
DSPARK_MODEL
points at ...-0731
, not the leftover -DSpark
string.
- Boot markers present: KV cache size, concurrency figure, “Application
startup complete”.
- Smoke test returns exactly
NVFP4 DSPARK OK
.
- Patch checks pass:
is_prefill_chunk
count 5, shared_experts
rows
present (Patch 4).
- Benchmarks run with
stream:false
and after warm-up. #
capture_runtime.sh
called with the explicit lowercase container name.
The deployment is real, it matches the guide’s headline numbers (84 tok/s peak, ~67 mean decode, ~220+ aggregate at c6, ~2.7K tok/s prefill at 100K), and the model is the same one you get from stock vLLM on the RTX6000 box, with one documented letter-counting flake on the custom fork’s sampler.
40-minute mixed-traffic soak
After the benchmark table above, we ran the guide’s realistic-mixed-traffic soak
to see how the deployment holds up under sustained agent-shaped load. It
reproduced the guide’s numbers within ~2%.
- This run: 532 requests / 209,566 completion tokens / 25,414 prompt tokens /
40 min at concurrency 4, stream:false.
- Aggregate 87.2 tok/s (guide 88.6, -1.6%), per-stream 21.8 (guide 22.3,
-2.2%). 0 errors, 0 soft-empty, 0 degenerate.
- Per-stream workers 21.24 / 22.27 / 22.04 / 21.81. Latency wall p50 15.9s,
p90 42.1s, p95 44.8s, p99 47.2s, max 52.1s.
Method
The guide’s own benchmarks/soak.py
is the exact agent-shaped soak behind the
guide’s 553 / 212,974 / 88.6 numbers. We ran an enhanced copy
(soak_enhanced.py
) with identical 6 prompt types (tool/code/JSON/reasoning/
long-context/chat), identical mixed temps (0.0/0.3/0.7) and budgets
(300/500/800), concurrency 4, stream:false, 40 min. We warmed the engine first
(guide’s bench_full.warm
: 5 long + 4 short) since warm state decays after
~30 min idle.
Server health
No OOM, no preemptions, no errors, no restarts on either node; only 1 benign
Triton JIT-compile warning at the soak’s first second; both vllm containers
Up 3h, never restarted.
Artifacts at /nfs/Dual_Spark_DSV4Flash0731/soak/
(soak_enhanced.py
,
soak-0731-2xspark.json
/.jsonl
/.md
, soak-calibrate.json
/.jsonl
).
Caveat
These are mixed-traffic numbers. The same server does 151 agg / 38.7 per-stream on the easy BST benchmark prompt and ~84 peak decode on toy content. ~87/22 is what realistic agent work gets.
Expanded coherence validation: 100 prompts
The 15/16 result was interesting, so we decided to do another ~100 prompts for more validation.
The harness is at /nfs/Dual_Spark_DSV4Flash0731/comparison100/
(100 prompts, 15 categories, staged and self-tested before the soak, run after the soak finished). The 16 original prompts are included byte-for-byte for continuity, and counting_traps
was deliberately expanded to 13 prompts to stress the count_letters
flake.
Result
83 same / 17 diff / 0 review / 0 fail. The verdict: the original
count_letters
flake was the visible tip of a SYSTEMIC counting/consistency deficit on the Spark deployment. Full is correct on all of these.
Per-category rollup:
| Category | same | diff | | counting_traps | 3/13 | 10 | | deterministic_math | 8/13 | 5 | | logic_puzzles | 4/5 | 1 | | reasoning_mode | 3/4 | 1 | | code_generation | 12/12 | 0 | | code_reasoning | 5/5 | 0 | | factual | 12/12 | 0 | | structured_output | 6/6 | 0 | | traps | 6/6 | 0 | | translation | 6/6 | 0 | | summarization | 4/4 | 0 | | creative | 4/4 | 0 | | long_reasoning | 4/4 | 0 | | multiturn | 3/3 | 0 | | ambiguity | 3/3 | 0 |
The counting traps (Spark wrong, Full correct)
count_letters
5 vs 3, cnt_mississippi
6 vs 4, cnt_rhubarb
5 vs 2,
cnt_banana_a
6 vs 3, cnt_supercali_l
9 vs 3, cnt_supercali_i
9 vs 7,
cnt_letter_in_sentence
9 vs 4, cnt_substr
6 vs 3, cnt_vowels
6 vs 3,
cnt_triple
6 vs 3. Spark is only correct on the three that do not require
counting repeated letters (word-count, string reversal, palindrome).
Non-counting diffs: math_geom_pyth
6 vs 5 (hypotenuse 3-4), math_word_consec
45 vs 13, math_word_work
1.2 vs 2.4, logic_scheduling
6 vs 3, reason_mode_letter
11 vs 3.
Determinism (5x temp-0 re-runs)
Spark is non-deterministic on count_letters
({9:4, 5:1}) and math_word_work
({1.2:4, 2.4:1}); wrong-but-stable on the other wrong counts. Full is deterministic and correct. Two distinct Spark symptoms: (a) a systematic counting/arithmetic-consistency deficit, (b) sampler non-determinism at temp 0 on at least 2 tasks.
Artifacts
math_prob_dice
: Spark is correct (writes \frac{1}{6}
in LaTeX); the
fraction scorer’s literal-match check is a false negative.
math_word_trains
: flawed prompt (Train A 60mph leaves earlier AND faster, never catches up); the expected ‘40’ is wrong.
- Full’s
None
on 2 supercali prompts is the 512-token reasoning-budget trap; re-run at 4096 returns the correct 7 and 3.
Everything else is identical: code 17/17 (12 HumanEval all unit tests pass on both, plus 5 code-reasoning), factual 12/12, structured output 6/6, traps 6/6, translation 6/6, summarization 4/4, creative 4/4, long reasoning 4/4 (train_meet
10:54 AM solved identically), multiturn 3/3, ambiguity 3/3.
Bottom line
Same model, same weights, but the custom vLLM fork on 2x DGX Spark shows a
genuine, systemic weakness on exact-counting/consistency and trivial arithmetic
that the stock-vLLM Full deployment does not. The flake was not isolated.
Reproduction
Harness at comparison100/
(prompts.py
/json
,
compare_harness.py
, analyze.py
, report.py
, run_all.sh
, README.md
,
out/
with records.json
, results.json
, comparison_table.md
,
results_summary.md
, raw/
200 files, determinism_counting.json
determinism_math.json
).
From the git clone:
cd comparison100 && ./run_all.sh
Project task: one-page Breakout clone (event-driven JavaScript)
A single 2,500+ token “project” prompt was devised and given to both models.
The goal was a real, end-to-end software-engineering task rather than short
prompts: build a one-page HTML5 Breakout clone that doubles as a teaching
example of event-driven JavaScript for a novice. The prompt specified project
organization, code layout, comment style, art/graphics (exact hex palette,
8x6 brick grid), game mechanics, an event-driven teaching angle (CustomEvent
pub-sub, event delegation, event.code), a deliberate twist (combo multiplier +
phasing every 5 bricks), Web Audio, localStorage high score, and a strict
quality bar. The full prompt is shared at
/nfs/Dual_Spark_DSV4Flash0731/project_task/prompt.md
(~4,480 words).
Stats
| Metric | Spark (2x DGX) | Full (4x RTX6000) | | Elapsed | 229.7 s | 32.6 s (7x faster) | | prompt_tokens | 5,807 | 5,807 | | completion_tokens | 13,432 | 10,303 | | Output size | 67,444 B / 1,551 lines | 45,779 B / 1,137 lines | | finish_reason | stop | stop | | Reasoning field | none | 1,630 chars | | Layout chosen | single-file index.html | single-file index.html |
Spec coverage
Spark 33/38 requirements met (5 partial, 0 missing), playable end-to-end.
Full 22/38 (5 partial, 11 missing), NOT playable. Both passed node --check
(valid JS) and HTML balance. Review used a headless node vm sandbox with
DOM/canvas stubs to empirically verify runtime behavior, not just static
inspection.
The twist (combo/phasing)
CORRECT in BOTH artifacts, verified by scripted playthroughs. Combo
increments, multiplier = min(5, 1 + floor(combo/5)), phasing triggers at
combo % 5 === 0, resets on miss, 3-second timer expiry, paddle-hit
termination, lives interaction all per spec. No off-by-one, no wrong reset.
Notable: the Spark deployment’s earlier systemic counting deficit (10/13
counting prompts wrong) did NOT recur in this code-generation task. Minor
nits: Full dispatches score-update with a stale combo (before increment);
neither implements the spec’s phase countdown meter.
Full model’s critical bugs (why it is not playable)
- F-1 Critical: initAudio() is dead code, never called, so no sound ever plays.
- F-6 Critical: drawParticles() called without its ctx argument, so render()
crashes on the first brick break (ctx.globalAlpha on undefined).
- F-7 High: soft-lock after a miss, no relaunch path.
- F-2 High: win unreachable, no max level, setState(‘win’) never called.
- F-3 High: unprotected localStorage.setItem crashes in private mode.
- F-4 High: two-brick same-frame double-flips the axis, ball tunnels through
a brick (the exact bug the spec warned about).
- F-5 Med: level-complete any-key restarts the game instead of advancing.
- F-8 Med: no IIFE, global scope pollution.
Spark model’s bugs (all minor)
Dead lastHitPaddle field; playSound(‘levelcomplete’) never called; no
combo-milestone beep; score-update CustomEvent missing bubbles:true; no dedicated corner-stuck detector.
Direct comparison / verdict
Spark produced a more complete, more correct, better-organized artifact (IIFE + single state + setState), with roughly 2x the comment density (404 vs 220 comment lines, all WHY-style teaching comments).
The 67 KB vs 46 KB size delta is that comment density, i.e. Spark did more of what the spec asked, not padding. Both used the exact 12-hex palette, 8x6 grid, event.code, event delegation, visibilitychange, variable timestep and everything else in the spec document. Teaching sections: Spark 385 words, Full 286, both within the 200-400 word spec.
Overall verdict: on a long structured software-engineering task, the 2x Spark deployment with the Dell Pro Max with GB10 (custom vLLM fork) produced the better artifact, slightly, despite Full being 7x faster.
This inverts the short-prompt pattern: the fork’s counting/consistency deficit appears task-specific (exact counting and arithmetic in short-form), but not a general capability gap. I was surprised by this.
Full’s artifact, while faster to produce, wasn’t runnable without some minor fixes.
UPDATE: There may be a minor problem with vLLM on the Full 4x RTX6000 setup. I am still investigating that part.
Artifacts
TODO
Follow-Up
Required Reading: Why your local LLM feels dumber than it is
for follow up I’m working on, The rabbit hole on differences beween GH200, sm120 RTX Pro Blackwell and NvFP4 spark has been interesting to say the least.