Prefill a 284B model on Nvidia. Decode it on Apple Silicon. Over plain 10GbE A prefill/decode disaggregation setup running DeepSeek-V4-Flash — a 284B total / 13B active model with 256 routed experts — bridged a 700,630-token cold prompt end to end in 11 minutes 26 seconds over plain 10 gigabit Ethernet, with 342 blocks and 1,117 tok/s of prefill, according to results published by the pd-bridge project. Prefill ran on 2× NVIDIA DGX Spark (GB10) via vLLM TP2 in FP8 while decode ran on a single Mac Studio M3 Ultra via oMLX in MXFP4, with the bridge computing the decoder's finished cache on the prefill machine rather than transferring a KV cache between incompatible formats. The bridged leg scored 5/5 on the judged quality eval, matching native, and the served window has grown from 262,144 to 2,097,152 tokens, though the project reports no Mac-alone control at these sizes and measured ratios stop at 241K tokens (3.7x). Prefill a 284B model on NVIDIA. Decode it on Apple Silicon. Over plain 10GbE. Two production inference engines that share no cache format, no framework, no vendor and no quantization, serving one request together. DeepSeek-V4-Flash is 284B total / 13B active, 256 routed experts, MLA + sparse attention, 149 GB resident on the prefill side and 156 GB on the decode side: - Prefill: 2× NVIDIA DGX Spark GB10 , vLLM TP2, official deepseek-ai/DeepSeek-V4-Flash FP8 - Decode: 1× Mac Studio M3 Ultra, oMLX, DV4-Flash-MXFP4-MLX MXFP4 - Link: ordinary 10 gigabit Ethernet. No RDMA, no Thunderbolt. php cold prompt Mac Studio alone Sparks prefill - Mac decode ~25K tokens 42.6 s 28.2 s 1.5x ~82K tokens 205.8 s 72.9 s 2.8x ~105K tokens 245.6 s 75.5 s 3.3x ~241K tokens 732.3 s 200.3 s 3.7x decode rate unchanged 23-25 tok/s both ways ; warm turns bypass the bridge 4.9 s / 19.3 s at 241K Every row verdict-checked, 2026-09-06. The bridged leg scores 5/5 on the judged quality eval , same as native. Since then the served window went 262,144 - 2,097,152 and the ceiling moved with it. A 700,630-token cold prompt now bridges end to end in 11 minutes 26 seconds — verdict complete , 342 blocks, 1,117 tok/s of prefill. That is 2.9x past the largest prompt in the table above , and the pair has accepted over a million. cold prompt engine tok/s end-to-end blocks verdict 192,099 119.2 s 1,611 143.1 s 93 complete 385,838 288.7 s 1,337 466.6 s 188 complete 677,069 596.8 s 1,134 655.9 s 330 complete 700,630 627.0 s 1,117 685.5 s 342 complete 709,055 635.4 s 1,116 694.6 s 345 partial 706,560/709,055 99.6% 988,487 1,037.8 s 952 1,142.5 s 376 partial 770,048/988,487 78% 1,006,172 1,070.5 s 940 1,171.7 s 346 partial 708,608/1,006,172 70% All 2026-09-08, one prefill pair, same 10GbE. partial is not a failure: past roughly 772K tokens the capture crosses the prefill box's free-memory floor and seals a valid contiguous prefix instead of dying; the decoder mounts what arrived and natively prefills the remainder. See Known limits . We have not run a Mac-alone control at these sizes , so those rows carry no ratio — they are what the bridge does, not a claimed speedup. The measured ratios stop at 241K and are shown above. Full numbers and methodology: RESULTS.md https://github.com/chadhurley25075-png/pd-bridge/blob/main/RESULTS.md · bench/BENCHMARK-PROTOCOL.md https://github.com/chadhurley25075-png/pd-bridge/blob/main/bench/BENCHMARK-PROTOCOL.md Prefill/decode disaggregation is well established, and so is the hardware argument for it: prefill is compute-bound, decode is memory-bandwidth-bound, so run each phase where it is cheapest. Existing systems do this by transferring the KV cache from the prefill worker to the decode worker. That requires both ends to agree on a cache format. Ours never can. One side is CUDA/vLLM with an FP8 paged cache; the other is Metal/MLX with its own block layout. Worse, DeepSeek-V4-Flash does not have "a KV cache" — each layer carries a rotating 128-token window, a compressor pool ratio 4 with overlap carry, and ratio 128 , and an indexer pool, all with layer-dependent RoPE. So we don't transfer a cache. We compute the decoder's finished cache on the prefill machine, using the decoder's own weights, and write it straight into the decoder's prefix-cache store. The prefill engine already computes the exact tensor those pools are a pure function of — the attention input. A hook takes it there, applies the Mac's projection and pooling math on the GPU, and emits the finished pools. The decode side assembles them into MLX cache objects and hands them to oMLX's own block writer. oMLX then sees a normal prefix-cache hit and only decodes. Neither engine is modified in its hot path; the decoder does not know a bridge exists. Payload: ~10 KB per token — 0.80 GB for an 81K-token prompt, pulled in 1.08 s. The network stopped being the bottleneck; the prefill engine is now 65% of wall time, which is where you want it. The whole design rests on the pooled tensors being the same tensors the decoder would have computed. That is tested, not assumed: | check | result | |---|---| | Cache arrays rebuilt on the Mac vs. a full native forward | 313/313 bit-exact | | Blocks written by the bridge vs. blocks oMLX writes itself | 11/11 identical only the created at stamp differs | | Torch pooling port vs. MLX ground truth T=23,217 | projections, window, carries bit-exact ; pooled tensors 99.95–99.96% identical, worst delta one bf16 ulp | | In-container hook selftest chunked == one-shot | 52/52 | | Needle retrieval through a fully reconstructed 81K cache | correct on every benchmark run | studio/verify blocks.py , studio/pd diff state.py , spark/pd pool validate.py and spark/pd pool selftest.py reproduce these. Compare tensors, never file hashes — created at means a bridge-written block can never be byte-identical as a file . The capture lives in the prefill box's memory for the whole request and costs about 11.8 KB of unified memory per token on rank 0 only rank 1 stays flat — it does its half of the attention and holds no capture . A GB10 has ONE 121 GB pool shared by weights, the vLLM KV arena and everything else, so steady-state free memory with the model up is ~22 GB. Measured: the capture crosses a 5.0 GB free-memory floor at roughly 772,000 tokens . Past that the hook seals a valid contiguous prefix 0,T and reports it rather than dying — the decoder mounts what arrived and natively prefills the tail. That is the partial verdict in the table above. A 700,630-token wake cleared the floor with 100 MB to spare ; a 1,006,172-token wake sealed at 70%. This replaced a real failure. Before the fix, finish copied every layer to the host without freeing the layers it had already written , so the full device capture and the growing host copies were alive at once. At 1,021,199 tokens it died after 11 of 43 layers and drove the box into swap thrash — no sshd even over a 200G fabric. It needed a physical power button. The hook now releases each layer as its file lands and checks MemAvailable every 64 layer-chunks. If you are memory-tight, this is your limit, not the window. The window is a config number; the floor is physics on your box. Measure MemAvailable during a long prefill before trusting either. The ~82K-token ceiling was a real bug. It is fixed. History kept because the failure mode is instructive and the arithmetic still matters on smaller machines. omlx block writer used to hold one materialised cumulative cache snapshot per 2048-token boundary until finalize : peak memory grew quadratically with prompt length ~ N N+1 /2 blocks' worth of arrays — ~16 GB at 39 boundaries, ~23 GB at 47, which exhausted a 256 GB M3 Ultra holding a 156 GB model and wrote zero blocks at 97,848 tokens . The writer now streams : each boundary is stored through oMLX's own pipeline and released the moment it is snapshotted begin stream / store boundary ; finalize drains and verifies . Peak memory is ONE boundary snapshot ~20 MB × boundary index / N — tens of MB, not tens of GB . It is validated against a real oMLX reference block synthetic layout match and live at 52 boundaries / 109,085 tokens — see RESULTS.md. PD STREAM BOUNDARIES=0 restores the batched path for comparison. PD MAX BRIDGE TOKENS remains as a configurable envelope guard, not a bug workaround: raise it to your machine's measured headroom. Two related behaviours worth knowing: - The fallback works, and it can no longer lie. When a bridge fails the reply still comes back correct — the decoder serves natively. Since the bench4 autopsy docs/FINDING-bench4-cold-fallback.md , every response carries an X-PD-Bridge verdict complete / partial B/T / declined with reason , and bench cold.py records it — a silent native fallback can never again enter a results table as a bridged number. - The "cold-start variance" at 20K was not variance. The 25.5 s vs 55.4 s spread was the capture hook flushing mid-request during chunked prefill see the FINDING : the 55 s runs were native fallbacks wearing a bridge label. The hook now guards its idle flush with a CUDA-event query and a chunk-alignment check, and the front validates every capture manifest before trusting it. This is a reference implementation, not a library. It is pinned hard and it is young. - One model. DeepSeek-V4-Flash. The pooling math is specific to its sparse attention. - Pinned stacks. oMLX 0.6.4; vLLM 0.21.1rc1 with the DeepSeek-V4 plugin sparkrun image . - It monkey-patches private internals of both engines — a sitecustomize hook onto DeepseekV4MultiHeadLatentAttentionWrapper.attention impl on the vLLM side, and a filesystem-fallback patch to oMLX's PagedSSDCacheIndex on the MLX side oMLX indexes SSD blocks at model load only, so externally written blocks are otherwise invisible . Expect this to break when either project moves. - The judged quality eval is five questions on one document. The bridged leg scores 5/5 on it, twice once from a fresh cold v3 bridge , same as native. Prefill runs FP8 weights and decode runs MXFP4, so bridged output is not token-identical to native; it is factually faithful on what we checked, which is a smaller claim than "equivalent". - The flush signal was unreliable until 2026-09-06 17:52 — fixed. The hook's watcher ran in three processes and two of them deleted the signal before the capturing worker saw it ~1 in 3 hit rate . Hook v5 fixes it; captures now close 0.6–0.9 s after the engine returns 4.8 s at 236K, which is the block write . Autopsy: docs/FINDING-flush-signal-three-watchers.md . Unit test: spark/test flush decision.py . - The front door is threaded, with one caveat. HTTP handlers run in threads health, model list and oMLX passthrough answer immediately, and concurrent decodes overlap because the decoder batches them , but every bridge call — the MLX cache assembly — is marshalled to the main thread and runs one at a time, because MLX streams are thread-local and the model lives there. Measured 2026-09-06: a short request completed in 47 s while an 86K-token cold bridge was in flight, instead of waiting it out. Two clients do slow each other down; they no longer block each other. - Only cold, long prompts benefit. Warm turns bypass the bridge by design and are served natively. The transferable idea is bigger than this code: when two engines cannot share a cache format, compute the consumer's finished cache on the producer , using the consumer's weights. That generalizes past this model and this hardware, and it is the part worth stealing. spark/ prefill side NVIDIA / vLLM capture sitecustomize v3.py the hook: projections + pooling on the GPU, per-layer safetensors pd pool torch.py torch port of the decoder's pooling math RoPE, compress, rmsnorm pd pool selftest.py in-container selftest chunked == one-shot pd pool validate.py validate the port against MLX ground truth pd-launch-v3.sh launch vLLM with the hook PD HOOK=off for a control run pd capture http.py Range-capable server so the decoder can stream captures pd share.py the threaded share the pooled path actually runs SimpleHTTP drops connections under poll+fetch pd-hf-layout.sh lay the checkpoint out as an HF hub dir inside the container mount POOL-VALIDATION.md what the validation numbers mean studio/ decode side Apple Silicon / oMLX pd front.py OpenAI-compatible front door; orchestrates a request end to end omlx block writer.py drive oMLX's own store pipeline to emit prefix-cache blocks pd assemble blocks.py build MLX cache objects from a pooled capture, snapshot per boundary pd export proj weights.py export the MLX projection weights the prefill hook needs pd export pool truth.py MLX-computed ground truth for validating the torch port pd make v3 from mlx.py build a v3 capture entirely in MLX acceptance harness pd capture mlx.py capture attention inputs natively test fixture pd rebuild mlx.py attention-only replay the v1 path, kept for comparison verify blocks.py directory-vs-directory block comparison pd diff state.py cache-array diff against a full forward test block writer synthetic.py pd omlx hooks.py kv/RDMA path: staged restore, tail install, restore timing PD OMLX HOOKS=1 pd-front-kv.sh / pd-rdma-recvd.sh front door in kv mode / the RDMA receiver pd assemble kv.py, pd verify kv.py, test omlx block.py kv-path assembly, bridged-vs-native check, block header gate spark/ kv/RDMA path pd kv connector.py vLLM v1 KV connector: oMLX-native blocks built on the GPU, pushed over RDMA during prefill pd omlx block.py oMLX chain hashes + block file header pd-launch-kv.sh, pd memguard.sh, pd-rdma-serve.sh, test kv gather.py rdma/ pd rdma receiver, R1 pull server/client + libpd rdma tx.so the connector's sender ; docs/RDMA.md bench/ bench cold.py records the X-PD-Bridge verdict , hetero the one-command demo client , BENCHMARK-PROTOCOL.md docs/ DESIGN-v3-pooled.md — the pooling math and the hook points, derived from oMLX's own code FINDING-bench4-cold-fallback.md — the mid-request-flush autopsy; what broke and what it taught FINDING-flush-signal-three-watchers.md — why the flush signal was consumed by the wrong worker FINDING-stale-limits-after-a-window-change.md — READ THIS BEFORE RAISING YOUR WINDOW. Five numbers sized against the old window that break silently after you raise it, including the one that clamps every long-context answer to a single token. Everything in this repo is written against the exact machines we ran, on purpose: if you have the same gear you get an exact replica and the numbers in RESULTS.md. If you don't, the idea is the same and the recipe scales down. Three rungs, honestly labeled: | rung | prefill side | decode side | model | status | |---|---|---|---|---| | A · exact replica | 2× DGX Spark, TP2 over their direct 200G cable | Mac Studio M3 Ultra 256 GB | DeepSeek-V4-Flash 284B / 13B active | measured — everything in RESULTS.md | | B · one Spark + any Apple Silicon Mac | 1× DGX Spark 128 GB | Mac Studio / Mac mini / iMac with ≥32 GB unified memory | a model that fits both boxes: the FP8 V4-Flash does not fit one Spark, so pick an MLA-latent model that does — DeepSeek-V2-Lite 16B is the obvious first | recipe only, unmeasured | | C · the kid's stack | one used CUDA gaming card 8–24 GB in a beat PC, vLLM or sglang | an M-series iMac / MacBook with 16 GB | the smallest MLA-latent model that fits both | recipe only, unmeasured | What is identical across all three rungs: the front door, the verdict header, the cold/warm decision, the block writer path, the benchmark protocol, the wire ordinary Ethernet — ~10 KB/token means even 1 GbE moves a 30K-token prompt in ~0.3 s . What changes when you move down: the model, and therefore the pooling math in the capture hook DeepSeek-V4-Flash's hook is specific to its sparse attention; a plain-MLA model like V2-Lite is simpler — its per-layer cache is the K/V rows themselves . docs/PORTING.md names the four seams you touch and has a two-question feasibility test that takes ten minutes. What to expect at the bottom rung, honestly: the win is the ratio of prefill speeds. A used 3090 prefills a 16B MLA model far faster than a 16 GB Mac does, so the shape of the result should hold; the absolute numbers will be smaller because the prompts and models are smaller. We have not run rungs B or C ourselves. They are the first ports we want to see, a negative result is a result, and we will feature whoever lands one. Open an issue. The point of this repo is that the privilege travels down. Take it apart. | side | hardware | software | model | |---|---|---|---| | prefill | 2× NVIDIA DGX Spark GB10, 128 GB each on a 200G RoCE link TP2 | Docker + the sparkrun vLLM image with the DeepSeek-V4 plugin aidendle94/sparkrun-vllm-ds4-gb10:production-ready , vLLM 0.21.1rc1 | deepseek-ai/DeepSeek-V4-Flash official FP8 checkpoint, ~149 GB | | decode | 1× Mac Studio M3 Ultra, 256 GB | oMLX 0.6.4 in a venv + the one-file patch in studio/ | an MLX MXFP4-experts / MXFP8-attention conversion of deepseek-ai/DeepSeek-V4-Flash-0731 ~156 GB; any bit-exact conversion works — ours keeps the DSpark MTP heads | | link | any Ethernet ≥10 GbE between the two | SSH key from the Mac to the prefill head; Python 3.10+ on both | — | The official FP8 checkpoint is ~149 GB, so it does not fit one 128 GB Spark: prefill is tensor-parallel across two Sparks over their direct ConnectX-7 link the standard two-Spark cable — box to box, no switch involved . The only traffic that crosses to the Mac is HTTP over ordinary Ethernet, through whatever switch you have. The numbers in RESULTS.md are TP2 over a 10 GbE LAN. The Mac-side model. We run a local, bit-exact MLX conversion of deepseek-ai/DeepSeek-V4-Flash-0731 MXFP4 experts, MXFP8 attention, DSpark MTP heads kept . There is no single published id to point at, so produce your own; the closest one-liner is mlx lm.convert --hf-path deepseek-ai/DeepSeek-V4-Flash-0731 --mlx-path ~/models/DV4-Flash-MXFP4-MLX \ -q --q-mode mxfp4 --q-bits 4 --q-group-size 32 unverified by us end to end — our build was a mixed conversion . What matters for the bridge is self-consistency , not which conversion: make weights exports the attention-projection weights from your MLX model, and the prefill hook uses exactly those, so the pooled tensors match whatever the decoder actually runs. cp config.example.env config.env && $EDITOR config.env nothing has a working default source config.env 1. Export the decoder's projection weights on the Mac, in the oMLX venv . These are what the prefill hook uses, so that the pooled tensors match the decoder's arithmetic rather than the prefill engine's: $OMLX PYTHON studio/pd export proj weights.py --model "$PD MODEL" --out "$PD V3" Copy $PD V3 pd pool torch.py , dv4 proj weights. , capture sitecustomize v3.py to both prefill nodes. 2. Start the prefill pair rank 0 = TP head, rank 1 = worker : ./spark/pd-launch-v3.sh 1 worker first ./spark/pd-launch-v3.sh 0 then head python3 spark/pd share.py "$PD CAPTURE DIR" 8010 pooled mode the one the numbers use or: python3 spark/pd capture http.py --root "$PD CAPTURE DIR" --port 8010 Range-capable; needed by the older 'hidden' pipelined mode 3. Patch and start oMLX , then the front door on the Mac : oMLX must notice blocks written after model load — one small patch, applied once, in the oMLX venv: OMLX PKG=$ $OMLX PYTHON -c 'import omlx,os;print os.path.dirname omlx. file ' .../site-packages/omlx patch -p0 -d "$OMLX PKG/cache" < "$OLDPWD/studio/omlx-0.6.4-paged ssd cache-disk-index-fallback.patch" then re start oMLX serving $PD MODEL on :8011, and start the front door: $OMLX PYTHON studio/pd front.py listens on $PD PORT 8012 , OpenAI-compatible curl -s localhost:8012/health {"ok": true, "front": "pd", ...} Point any OpenAI-compatible client at :8012 . Prompts under PD MIN TOKENS or with fewer than PD MIN TAIL uncached tokens go straight to oMLX; longer cold prompts are prefilled on the Sparks. The X-PD-Bridge response header says which happened. make doctor checks every link in the chain. 4. Benchmark: python3 bench/bench cold.py --chars 330000 --seed 301 --url http://