{"slug": "show-hn-deepseek-v4-flash-with-7-7-gib-ram-using-nvme-demand-paging", "title": "Show HN: DeepSeek V4 Flash with 7.7 GiB RAM using NVMe demand paging", "summary": "Mutaz Abubaker released a research preview of DeepSeek V4 Flash, a 78.62 GiB GGUF model with 284.33B logical parameters, that runs on a Linux laptop with only 7.7 GiB of physical RAM and no GPU using mmap-backed NVMe demand paging. The v0.1 preview achieves a controlled cold first-token diagnostic of 5.33 seconds with zero process swaps and a maximum RSS of 6,204,632 KiB, but does not claim the full model fits in RAM or production-grade speed. The fork of antirez/ds4 supports CPU-only execution of AProjQ4 quantized GGUFs and is credited to research by Mutaz Abubaker with assistance from ChatGPT (GPT-5.6 Sol).", "body_md": "**Run a 78.62 GiB DeepSeek V4 Flash GGUF on a Linux laptop with 7.7 GiB of physical RAM and no GPU, using mmap-backed NVMe demand paging.**\n\nImportant\n\n**The model does not fit entirely in 8 GB of RAM.**\nThe GGUF remains file-backed on NVMe and Linux faults model pages into memory on demand. This repository demonstrates execution under a severe model-size-to-RAM mismatch; it does not claim that 78.62 GiB of weights are resident in physical memory.\n\n| Metric | Result |\n|---|---|\n| Model | DeepSeek V4 Flash |\n| GGUF size | 78.62 GiB |\n| Logical parameters | 284.33B |\n| Physical RAM | 7.7 GiB |\n| Backend | CPU only |\n| GPU / CUDA | None |\n| Controlled cold first-token diagnostic | 5.33 s |\n| Maximum RSS | 6,204,632 KiB |\n| Process swaps | 0 |\n| Storage path | mmap-backed NVMe demand paging |\n\n**DeepSeek V4 Flash AProjQ4 CPU support** for GGUFs whose dense attention projections are stored as`Q4_K`\n\n.**Opt-in CPU demand paging** for systems where the GGUF is much larger than physical RAM:\n\n```\nexport DS4_CPU_NO_PREFETCH=1\n```\n\n- CPU-only execution of the tested DeepSeek V4 Flash GGUF under extreme memory pressure.\n- A model file much larger than installed physical RAM can execute through file-backed demand paging.\n- The controlled public first-token diagnostic completed without process swap.\n- The repository includes methodology, benchmark data, evidence tracking, and explicit claim boundaries.\n\n- That the complete 78.62 GiB model fits inside 7.7 GiB RAM.\n- Production-grade inference speed.\n- Competitive sustained generation throughput.\n- That the 5.33-second diagnostic equals normal chat latency.\n- That Linux VM tuning results generalize to other machines or storage devices.\n- Universal support for every DeepSeek GGUF or quantization.\n\n**v0.1 Research Preview**\n\nThe public v0.1 claim is intentionally narrow: CPU-only execution of an AProjQ4 DeepSeek V4 Flash GGUF on a memory-constrained Linux system using NVMe-backed demand paging.\n\nCurrent follow-up work focuses on sustained decode behavior, page faults and refaults, Linux VM pressure, swap policy, and expert-weight I/O.\n\n— measurements, evidence classifications, hashes, and claim boundaries.`docs/evidence-register.md`\n\n— model and test scope.`MODEL_CARD.md`\n\n— benchmark data.`benchmarks/`\n\n— release and upstream QA context.`QA_BEFORE_RELEASES.md`\n\nBuild the CPU target:\n\n```\nmake cpu\n```\n\nEnable the memory-constrained demand-paging path:\n\n```\nexport DS4_CPU_NO_PREFETCH=1\n```\n\nThen use a supported DeepSeek V4 Flash GGUF with the normal DwarfStar CLI workflow documented below.\n\nThis is a research fork of [ antirez/ds4](https://github.com/antirez/ds4).\n\nThe upstream project, architecture, inference engine, and original documentation remain credited and preserved below.\n\nResearch, integration, benchmarking, and release work: **Mutaz Abubaker**.\n\nResearch and port assistance: **ChatGPT (GPT-5.6 Sol)**.\n\nThe original upstream README is preserved below.\n\n**DwarfStar** is a small native inference engine optimized first for\n**DeepSeek V4 Flash**. It also supports **GLM 5.2** and, on very high-memory\nmachines, **DeepSeek V4 PRO**. It is self-contained and deliberately narrow,\nnot a general GGUF runner. Model loading, prompt rendering, tool calls, KV\nstate, the HTTP server, and the coding agent are built and tested together.\nThe repository also includes tools and data for GGUF, imatrix, quality, and speed.\n\nSupported backends:\n\n**Metal**, the primary target, on Macs with 96 GB or more. Smaller machines can use SSD streaming.** NVIDIA CUDA**, including multi-GPU systems and DGX Spark.** ROCm**on Strix Halo systems such as the Framework Desktop.\n\nThis project would not exist without **llama.cpp and GGML**, make sure to read\nthe acknowledgements section, a big thank you to Georgi Gerganov and all the\nother contributors.\n\nModel support is intentionally opportunistic. The project follows the best open weights for useful local machine sizes, especially 128 GB laptops and 512 GB workstations. A model may be removed when a better replacement arrives.\n\n- You can run a very capable models in your consumer hardware, a MacBook, a DGX Spark, or a Strix Halo for example. Even if you have not enough RAM, with SSD streaming, you can run it at a decent speed.\n- Using the CUDA multi-GPU support and with ds4-server micro batching of decoding and generation, you can turn a server with old-ish CUDA cards (Ada Lovelace architecture), no longer supported for new models by vLLM, into a multi-user LLM server for your company. We tested this setup with 8xL40S NVIDIA cards and multiple sessions with very good results. 120 t/s aggreated generation, 2000 t/s prefill.\n- Using two MacBook M5 Max / M3 Ultra RDMA, you can run 4 bit DeepSeek Flash or GLM 5.2 with tensor parallelism.\n- You can also use pipeline paralellism to glue together multiple systems to sum their RAM and run larger models.\n\n- Capable open-weight models now fit on high-end personal machines.\n- DeepSeek V4 Flash and PRO, GLM 5.2, tolerate aggressive routed-expert quantization.\n- Compressed KV caches and fast local SSDs make long contexts practical.\n- The idea of an inference system specialized for a few models.\n\n- This software is developed with\n**strong assistance from GPT 5.5, 5.6, Claude Fable** and with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. The acknowledgement below is equally important: this would not exist without`llama.cpp`\n\nand GGML, largely written by hand.\n\n`ds4.c`\n\ndoes not link against GGML, but it **exists thanks to the path opened by the\nllama.cpp project and the kernels, quantization formats, GGUF ecosystem, and hard-won\nengineering knowledge developed there**.\nWe are thankful and indebted to [ llama.cpp](https://github.com/ggml-org/llama.cpp)\nand its contributors. Their implementation, kernels, tests, and design choices were\nan essential reference while building this DeepSeek V4 specific inference path.\nSome source-level pieces are retained or adapted here under the MIT license: GGUF\nquant layouts and tables, CPU quant/dot logic, and certain kernels. For this\nreason, and because we are genuinely grateful, we keep the GGML authors copyright\nnotice in our\n\n`LICENSE`\n\nfile.The software is currently very fast changing. Consider it beta quality. Before each release, a big QA run is executed, however instabilities are definitely possible.\n\nI (Salvatore) believe that the way projects should be shipped and used changed because of AI. The main differences today are:\n\n- With AI, users can modify the software in significant ways with low efforts, costs, and even lacking deep domain knowledge about the task they want to accomplish. For instance, a DwarfStar user with a specific hardware setup can ask a coding agent to improve the inference speed of this software for the specific hardware setup, asking the model to reach the maximum prefill and generation speed without impacting correctness, and also asking to do a deep QA pass.\n- Similiarly, because of \"1\", software may be shipped in a different way than before. It must be more a working template for the biggest use cases, without trying to cover every possible setup. If DwarfStar showcases a few good implementations of tensor parallel execution, the code will work as a rail for implementing the same feature in specific conditions, for a new model, and so forth.\n\nSo, while this project attempts to be usable for the featured models and the most common hardware setups, I ask you, if you have access to coding agents, to consider using coding agents as an interface to discover the project, make modifications, create personalized setups. This way you can likely do more than what we ship, and certain things that are not documented or implemented, and that you require, are potentially very easy to achieve.\n\nIf you are looking for very specific things, we have other sub-README files. Otherwise for normal usage keep reading the next sections.\n\n[CONTRIBUTING.md](/baker27727/ds4-8gb-cpu/blob/main/CONTRIBUTING.md): correctness and speed regression testing guide for contributors.**Read this before sending a pull request**.[QA_BEFORE_RELEASES.md](/baker27727/ds4-8gb-cpu/blob/main/QA_BEFORE_RELEASES.md): the complete release test matrix, including the remote Metal, CUDA, and ROCm machines.[gguf-tools/README.md](/baker27727/ds4-8gb-cpu/blob/main/gguf-tools/README.md): offline GGUF generation, imatrix collection, quantization tooling, and quality checks.[gguf-tools/imatrix/README.md](/baker27727/ds4-8gb-cpu/blob/main/gguf-tools/imatrix/README.md): how the routed-MoE imatrix is collected and used.[gguf-tools/imatrix/dataset/README.md](/baker27727/ds4-8gb-cpu/blob/main/gguf-tools/imatrix/dataset/README.md): how the calibration prompt corpus is generated.[gguf-tools/quality-testing/README.md](/baker27727/ds4-8gb-cpu/blob/main/gguf-tools/quality-testing/README.md): how local GGUFs are scored against official DeepSeek V4 Flash/PRO continuations.[dir-steering/README.md](/baker27727/ds4-8gb-cpu/blob/main/dir-steering/README.md): directional steering data, vector generation, and usage.[speed-bench/README.md](/baker27727/ds4-8gb-cpu/blob/main/speed-bench/README.md): benchmark commands, charts, and CSV generation.[tests/test-vectors/README.md](/baker27727/ds4-8gb-cpu/blob/main/tests/test-vectors/README.md): official continuation vectors used for regression checks.\n\nThis implementation only works with the DeepSeek V4 and GLM 5.2 GGUFs listed below. It is not a general GGUF loader, and arbitrary GGUF files will not have the tensor layout, quantization mix, metadata, or optional MTP state expected by the engine. The 2 bit quantizations provided here are verified to be actually high quality: they behave well, work under coding agents, call tools in a reliable way.\n\nThe 2 bit quants use a very asymmetrical quantization: only the routed MoE\nexperts are quantized, up/gate at `IQ2_XXS`\n\n, down at `Q2_K`\n\n. They are the\nmajority of all the model space: the other components (shared experts,\nprojections, routing) are left untouched to guarantee quality.\n\nDownload one main model. **Prefer the imatrix versions.**\n\n```\n./download_model.sh ds4f-q2      # 96/128 GB RAM machines\n./download_model.sh ds4f-q2-q4   # q2 with the last 6 expert layers at q4\n./download_model.sh ds4f-q4      # >= 256 GB RAM machines\n./download_model.sh ds4f-mxfp4   # native MXFP4 experts, about 156 GB\n./download_model.sh pro-q2-imatrix  # 512 GB RAM machines, PRO q2 imatrix quant\n```\n\nThe MXFP4 GGUF preserves DeepSeek's released MXFP4 routed-expert weights rather than requantizing them. It runs on Metal and CUDA; Blackwell CUDA devices use native FP4 matrix instructions and FP4 activations for batched expert work. Decode and other CUDA devices use Q8 activations.\n\nFor the full PRO Q4 distributed run, download one half on each machine:\n\n```\n./download_model.sh pro-q4-layers00-30      # first half of PRO Q4 split\n./download_model.sh pro-q4-layers31-output  # second half of PRO Q4 split\n```\n\nThe script downloads from `https://huggingface.co/antirez/deepseek-v4-gguf`\n\n,\nstores files under `./gguf/`\n\n, resumes partial downloads with `curl -C -`\n\n, and\nupdates `./ds4flash.gguf`\n\nto point at the selected main model.\nThe `pro-q4-layers00-30`\n\n, `pro-q4-layers31-output`\n\n, and `pro-q4-split`\n\ntargets\ndownload distributed PRO Q4 pieces and do not update `./ds4flash.gguf`\n\n.\nAuthentication is optional for public downloads, but `--token TOKEN`\n\n,\n`HF_TOKEN`\n\n, or the local Hugging Face token cache are used when present.\n\nIf you want to regenerate GGUF files or collect a new imatrix, see\n[gguf-tools/README.md](/baker27727/ds4-8gb-cpu/blob/main/gguf-tools/README.md). Those tools are meant for offline\nmodel-building work and can take a long time on the full DeepSeek V4 Flash\nweights. Flash GGUF generation is supported by the local tools. PRO GGUF\nproduction currently still depends on the external `llama.cpp`\n\n-based workflow;\nnative tooling can be added later.\n\nGLM 5.2 support is limited to the GGUF files tested by this branch:\n\n```\n./download_model.sh glm-unsloth-q4  # Unsloth UD-Q4_K_XL, 11 shards\n./download_model.sh glm-antirez-iq2xxs  # antirez routed IQ2_XXS single-file GGUF\n./download_model.sh glm-antirez-q2  # antirez routed Q2_K single-file GGUF\n./download_model.sh glm-antirez-q4  # antirez routed Q4_K single-file GGUF\n```\n\nThe supported GLM layout keeps dense/model-control tensors in the existing\nQ8/F32 paths and supports routed expert gate/up tensors in `Q2_K`\n\n, `Q4_K`\n\n, or\n`Q5_K`\n\n; routed expert down tensors are supported in `Q2_K`\n\n, `Q4_K`\n\n, `Q5_K`\n\n, or\n`Q6_K`\n\n. Other GLM GGUF quant layouts should be treated as unsupported until they\nare added deliberately and scored against the official 100-case fixture.\n\nThese formats do not all support the same execution modes. The Q4 files work for normal Metal and CUDA inference. Two-Mac tensor parallelism currently requires an ownership-aware IQ2_XXS or Q2_K routed layout; a routed Q4 GLM must be rejected before evaluation.\n\nGLM's MTP block is part of the main GGUF; it does not use the separate Flash\nMTP file. Ordinary decode remains the default. `--glm-mtp`\n\nenables experimental\ngreedy speculation. `--glm-mtp-timing`\n\nalso enables it and prints acceptance\nand timing counters:\n\n```\n./ds4 -m gguf/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf \\\n  --glm-mtp-timing --temp 0\n```\n\nGLM inference uses the Metal, CUDA, or ROCm graph backend. Directional steering,\n`--power`\n\nbelow 100, an explicit `--prefill-chunk`\n\n, and the external `--mtp`\n\nfile are not supported for GLM yet.\n\nThen build:\n\n```\nmake                  # macOS Metal\nmake cuda-spark       # Linux CUDA, DGX Spark / GB10\nmake cuda-generic     # Linux CUDA, other local CUDA GPUs\nmake strix-halo       # Linux ROCm, AMD Strix Halo\nmake cpu              # CPU-only diagnostics build\n```\n\n`./ds4flash.gguf`\n\nis the default model path used by both binaries. Pass `-m`\n\nto\nselect another supported GGUF from `./gguf/`\n\n. Run `./ds4 --help`\n\nand\n`./ds4-server --help`\n\nfor the full flag list.\n\nDSpark is an auxiliary draft model released by DeepSeek for DeepSeek V4 Flash. It reads hidden states from the main model and proposes up to five future tokens. DwarfStar checks those proposals with the main Flash model and commits only the accepted prefix. The main model remains authoritative; a rejected or low-confidence suffix falls back to ordinary target decoding.\n\nThe possible gain is faster generation: when several proposed tokens are accepted, one target verification pass advances the stream by several tokens. It does not accelerate prefill, and the draft and verification work is not free. Predictable continuations, especially code, tend to benefit most; low-yield prompts can be no faster or even slower. DSpark is therefore still experimental and explicitly opt-in.\n\nAccepted proposals keep the state produced by the batched target verifier\ninstead of running the same tokens through one-token decode again. Both paths\nexecute the same inference graph, but floating-point operations are grouped in\na different order. A long greedy DSpark run may therefore diverge from a run\nwithout DSpark after an otherwise valid accepted block. This is not a reduced\nprecision or approximate-model mode; use ordinary decoding, `--quality`\n\n, or\n`--dspark-strict`\n\nwhen byte-for-byte reproducibility with one-token decode is\nrequired.\n\nThe DSpark checkpoint for Flash 0731 is packaged here as a separate support GGUF of about 5.6 GiB. It is not a standalone model. Download it once:\n\n```\n./download_model.sh ds4f-dspark\n```\n\nThe support file can be used with the 0731 Flash `ds4f-q2`\n\n, `ds4f-q2-q4`\n\n, and\n`ds4f-q4`\n\nmodels listed above. It is checkpoint-specific\nand must not be paired with an older Flash model. For now **DeepSeek V4 PRO**\nis not supported. On Metal, the main model may be resident or use\n`--ssd-streaming`\n\n; the support model still adds its own weights and runtime\nstate to the memory requirement. DSpark replaces the legacy one-stage MTP\nsupport model for that run rather than stacking with it.\n\nRun it with greedy decoding:\n\n```\n./ds4 -m ds4flash.gguf \\\n  --mtp gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \\\n  --dspark --temp 0\n```\n\n`--mtp`\n\nsupplies the support GGUF, while `--dspark`\n\nselects the DSpark runtime.\nThe default confidence threshold is `0.6`\n\non Metal and `0.7`\n\non CUDA and ROCm;\nit prunes suffixes that are unlikely to repay their verification cost.\n`--dspark-confidence 0`\n\nforces fixed five-token blocks and is intended for\ndiagnostics. Sampled decoding does not use DSpark proposals. `--quality`\n\nand\n`--dspark-strict`\n\nalso keep target-only decoding, which is useful for\nreproducibility checks.\n\nThe current q2 results use `ds4-bench`\n\nwith the standard *Promessi sposi*\ninput, 2048-token context steps, and 128 greedy generation tokens at every\nfrontier. Each prefill number is for the next 2048-token chunk. The complete\nsweeps are in [m5_max.csv](/baker27727/ds4-8gb-cpu/blob/main/speed-bench/m5_max.csv) and\n[gb10.csv](/baker27727/ds4-8gb-cpu/blob/main/speed-bench/gb10.csv).\n\n| Machine | Backend | Context | Prefill | Generation |\n|---|---|---|---|---|\n| MacBook Pro M5 Max, 128 GB | Metal | 2048 | 790.18 t/s | 39.35 t/s |\n| MacBook Pro M5 Max, 128 GB | Metal | 16384 | 572.53 t/s | 36.14 t/s |\n| MacBook Pro M5 Max, 128 GB | Metal | 32768 | 557.04 t/s | 34.36 t/s |\n| MacBook Pro M5 Max, 128 GB | Metal | 65536 | 398.50 t/s | 27.64 t/s |\n| DGX Spark GB10, 128 GB | CUDA | 2048 | 825.76 t/s | 18.05 t/s |\n| DGX Spark GB10, 128 GB | CUDA | 16384 | 872.44 t/s | 15.10 t/s |\n| DGX Spark GB10, 128 GB | CUDA | 32768 | 855.94 t/s | 14.43 t/s |\n| DGX Spark GB10, 128 GB | CUDA | 65536 | 822.98 t/s | 13.84 t/s |\n\nOlder measurements for machines and model variants not rerun in this pass are kept for reference. They used the earlier CLI prompt procedure and are not directly comparable with the table above.\n\n| Machine | Quant | Prompt | Prefill | Generation |\n|---|---|---|---|---|\n| MacBook Pro M3 Max, 128 GB | q2 | short | 58.52 t/s | 26.68 t/s |\n| MacBook Pro M3 Max, 128 GB | q2 | 11709 tokens | 250.11 t/s | 21.47 t/s |\n| Mac Studio M3 Ultra, 512 GB | q2 | short | 84.43 t/s | 36.86 t/s |\n| Mac Studio M3 Ultra, 512 GB | q2 | 11709 tokens | 468.03 t/s | 27.39 t/s |\n| Mac Studio M3 Ultra, 512 GB | q4 | short | 78.95 t/s | 35.50 t/s |\n| Mac Studio M3 Ultra, 512 GB | q4 | 12018 tokens | 448.82 t/s | 26.62 t/s |\n| Mac Studio M3 Ultra, 512 GB | PRO q2 | 32768 tokens | 138.82 t/s | 9.56 t/s |\n\nThe normal Metal path tries to make the model resident in GPU-addressable\nmemory. This is the fastest path and should remain your default when the model\nfits. DwarfStar also has an **SSD streaming** capacity mode on Metal and for\nGLM 5.2 on ROCm. In this mode the non-routed model weights stay resident, while\nrouted MoE experts are kept in an in-memory cache and loaded from the GGUF file\non cache misses.\n\nStreaming is not as fast as fitting the full model in RAM. It still needs memory for non-routed weights, KV cache, graph scratch, activations, and the routed expert cache. It is useful because routed experts dominate model size and modern Mac SSDs are fast enough to make cache misses tolerable. Long prefills can still be fast; generation is more sensitive to cache misses because every new token routes through experts again.\n\nStart with the automatic cache budget:\n\n```\n./ds4 -m ./ds4flash.gguf --ssd-streaming\n```\n\nIf startup reports that the expert cache is too large, or if you want to reserve more memory for context, set the routed expert cache explicitly:\n\n```\n./ds4 -m ./ds4flash.gguf --ssd-streaming --ssd-streaming-cache-experts 32GB\n```\n\nThe `32GB`\n\nvalue is a routed-expert memory budget, not a generic byte cache.\nDwarfStar first reserves headroom for the two full routed layers used by\noverlapped streaming prefill, then converts the remaining bytes to the number of\ndynamic cached experts that fit for the current GGUF. Explicit `NGB`\n\nbudgets may\nalso be capped after context/KV accounting so the backend working set stays out of\nthe slow pressure zone. A plain number such as\n`--ssd-streaming-cache-experts 4000`\n\nis different: it means exactly 4000 dynamic\nexpert slots, with no extra accounting. Non-routed weights, KV cache, graph\nscratch, and activations need additional memory. The automatic cache budget takes\n80% of the backend's recommended working set, subtracts non-routed weights, then\napplies the same routed-prefill headroom before sizing the dynamic cache. Leave\nthe hot expert preload enabled for normal use; use `--ssd-streaming-cold`\n\nand\n`--ssd-streaming-preload-experts N`\n\nonly for measurements.\n\nOn 64GB MacBooks, start with the 2-bit Flash GGUF and a moderate expert cache:\n\n```\n./download_model.sh ds4f-q2\n\n./ds4 \\\n  -m ./ds4flash.gguf \\\n  --ssd-streaming \\\n  --ssd-streaming-cache-experts 32GB \\\n  --ctx 32768 \\\n  --nothink\n```\n\nOn 128GB MacBooks, PRO q2 streaming is experimental but usable for inspection\nand occasional work when you accept slow generation. Start with `--nothink`\n\n:\n\n```\n./download_model.sh pro-q2-imatrix\n\n./ds4 \\\n  -m gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.gguf \\\n  --ssd-streaming \\\n  --ctx 32768 \\\n  --nothink\n```\n\nOn an M5 Max with 128GB of RAM, a short PRO q2 streaming decode benchmark found\nthe automatic budget best: it selected about `59GB`\n\nof routed expert cache.\nManual `64GB`\n\nto `75GB`\n\ncaches were close on that machine. Prefer the automatic\nbudget; if setting the cache manually on this class of machine, start around\n`48GB`\n\nto `64GB`\n\n, then increase only while the machine remains responsive and\nthe startup log shows the requested dynamic cache. Once the machine is stable,\nre-enable thinking with a conservative generation limit:\n\n```\n./ds4 \\\n  -m gguf/DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.gguf \\\n  --ssd-streaming \\\n  --ctx 32768 \\\n  --think \\\n  --tokens 1500\n```\n\nGLM 5.2 uses the same option. Its streaming path keeps the largest full-layer prefix that fits resident, then uses the remaining budget for a dynamic expert cache. Start with the automatic budget:\n\n```\n./ds4 \\\n  -m gguf/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf \\\n  --ssd-streaming \\\n  --ctx 32768\n```\n\nThe important startup line is the cache report. Start conservative, then increase the cache if the machine has headroom.\n\nOn a 128GB Strix Halo, use the routed Q2_K model and a 4096-token context as the starting point. The automatic cache budget leaves room for the GLM graph and KV state:\n\n```\n./download_model.sh glm-antirez-q2\nmake strix-halo\n./ds4 --rocm -m gguf/GLM-5.2-UD-Q2_K_RoutedQ2K.gguf \\\n  --ssd-streaming --ctx 4096\n```\n\nPipeline parallelism lets DwarfStar **run a model that is too large for one machine** by\nsplitting transformer layers across multiple machines. The main example is the\nfull 4-bit Flash quant across two 128 GB MacBooks: each process maps only its\nown layer slice, activations are sent over TCP, and the coordinator keeps normal\nCLI/API behavior.\n\nPipeline parallelism can also **speed up prefill** by\nusing multiple GPUs at the same time to process different micro-batches at\ndifferent layers, like in an assembly line. Only prefill can be accelerated this\nway. Generation is purely autoregressive: each token must finish across the\nroute before the next token can start. The model work is the same as a single\nprocess, plus coordination latency, so distributed generation is slower.\n\nTo build an initial mental model, here are the high level concepts:\n\n- You put the GGUF on every machine, but each one loads just a subset.\n`--layers`\n\ncontrols which tensors are mapped, so a worker with`--layers 20:output`\n\ndoes not load the earlier layers. - Layer ranges are inclusive:\n`10:20`\n\nmeans layers 10, 11, ..., 20.`N:output`\n\nmeans layer`N`\n\nthrough the final layer plus the output head. - You assign one of the machines the role of\n`coordinator`\n\n, the others the roles of`workers`\n\n. Workers will connect to the coordinator and will tell they are there and which layers they are able to process. - Each worker keeps its slice of the KV cache.\n- Communication is worker-to-worker, there is no need to use the coordinator as relay, so if your coordinator is\n`A`\n\n, and you make a request, activations will flow in`A -> B -> C -> back to A`\n\n.\n\nThe prefill path is pipelined (this is why it can go faster than in a single machine). For large prompts the coordinator can run its slice on chunk N+1 while the worker is running its slice on chunk N. The distributed rows below were measured with two M5 Max 128 GB MacBooks connected by Thunderbolt 5, using the Q4 Flash GGUF and the default 4096-token distributed prefill chunk. The single-process column is a reference run with the Q2 GGUF on a single machine, so it actually is a bit faster since the routed MoEs are smaller.\n\n| Prompt | Single-process reference | Two MacBooks | Speedup |\n|---|---|---|---|\n| 9421 tokens | 421.70 t/s | 582.22 t/s | 1.38x |\n| 28684 tokens | 405.30 t/s | 674.16 t/s | 1.66x |\n| 63819 tokens | 353.62 t/s | 654.79 t/s | 1.85x |\n\nGeneration is different. **It is strictly autoregressive**: token N+1 cannot start\nuntil token N has produced logits and sampling has selected the next token. That\nmeans distributed generation cannot use the long prefill pipeline. It pays at\nleast one cross-machine activation hop per generated token, so generation is\nslower than a single local process. On the same two-Mac Thunderbolt setup, a\n12k-context control run with the 91 GB Flash quant went from 30.59 t/s\nsingle-process to 24.67 t/s distributed, a 19.4% loss. Distributed inference is\ntherefore mainly for fitting larger models and speeding up long prefills, not\nfor making decode faster.\n\nThe full-size PRO Q4 GGUF can be run across two 512 GB Mac Studio M3 Ultra\nmachines by giving the coordinator layers `0:30`\n\nand the worker\n`31:output`\n\n. Use the split GGUF files so each side maps only the tensors it\nneeds:\n\n```\n# Coordinator machine.\n./download_model.sh pro-q4-layers00-30\n\n# Worker machine.\n./download_model.sh pro-q4-layers31-output\n```\n\nThe two files are:\n\n```\ngguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf\ngguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf\n```\n\nThis is a capacity use case: each process maps only its own half of the model, while the worker owns the output head and returns logits.\n\nThe current PRO Q4 Metal path uses queue-resident exact expert tables for the\nlarge routed experts. This avoids the broad multi-GiB routed-tensor bindings\nthat made early distributed PRO Q4 attempts either run very slowly or hit Metal\nmemory accounting limits. In a short greedy smoke test over the direct\n`192.168.0.182`\n\n/ `192.168.0.183`\n\nlink, the model generated coherent text and\nmeasured 11.47 t/s generation after startup. Per-token telemetry was balanced:\nlocal layers were around 39-43 ms, remote layers around 44-49 ms, for total\ntoken times around 84-92 ms. Expect a slow startup while each side maps and\nmakes its half of the model resident. Long-context PRO Q4 prefill and decode\nperformance still needs separate benchmarking.\n\nThe measurements above use a Thunderbolt 5 cable. The implementation is plain TCP and also works over slower links, including WiFi, but fast Ethernet or Thunderbolt networking is strongly recommended. Slow links mostly hurt generation latency and short prefills; large prefills can still benefit when the layer split is balanced. In the normal performance path, the last worker owns the output head and returns logits directly.\n\nMinimal two-host configuration:\n\n```\n# Machine A: coordinator, owns tokenization, sampling, the prompt, and layers 0..30.\n./ds4 \\\n  -m gguf/DeepSeek-V4-Pro-Q4K-Layers00-30.gguf \\\n  --role coordinator \\\n  --layers 0:30 \\\n  --listen 169.254.43.68 1234\n\n# Machine B: worker, connects to A and owns layers 31..output.\n./ds4 \\\n  -m gguf/DeepSeek-V4-Pro-Q4K-Layers-31-output.gguf \\\n  --role worker \\\n  --layers 31:output \\\n  --coordinator 169.254.43.68 1234\n```\n\nNormally the final worker should own the output head too, for example\n`--layers 20:output`\n\n. This avoids returning a full final hidden-state batch\nafter prefill and lets the final worker produce the logits directly. On very\nslow or metered links, `--layers 20:42`\n\nis also supported: the coordinator will\nload the output head and compute logits locally, trading extra coordinator work\nfor smaller per-token replies.\n\nThe table below shows the same two M5 Max hosts, the same 91 GB Flash quant,\ncoordinator `--layers 0:19`\n\n, worker `--layers 20:output`\n\n, an 8192-token prompt\nfrom `speed-bench/promessi_sposi.txt`\n\n, and 128 generated tokens. WiFi and\nInternet numbers vary with local conditions, but the shape is the important\npart: high latency hurts generation directly, while lower bandwidth also pulls\ndown long-prefill speed.\n\n| Link | Addresses | Ping avg | Prefill | Generation |\n|---|---|---|---|---|\n| Thunderbolt 5 | `169.254.43.68` -> `169.254.12.245` |\n0.45 ms | 582.99 t/s | 25.09 t/s |\n| WiFi | `192.168.1.57` -> `192.168.1.95` |\n77.20 ms | 250.70 t/s | 10.70 t/s |\n| Internet / VPN | `10.77.0.4` -> `10.77.0.3` |\n152.10 ms | 114.88 t/s | 3.63 t/s |\n\nThe Internet/VPN case is not meant to be a good interactive experience. It is still useful for collective testing: multiple people can temporarily combine machines to run a larger model that would not fit on any single host, accepting slow decode in exchange for being able to inspect the model at all.\n\nUse the coordinator exactly like normal `./ds4`\n\n: interactive chat, `/read`\n\n,\nand ordinary generation go through the same high-level session API. The same\ndistributed options are also wired into `ds4-agent`\n\n, `ds4-eval`\n\n, and\n`ds4-bench`\n\n. For benchmarks, workers should already be running; `ds4-bench`\n\nwaits until a complete route is available.\n\nUseful tuning and diagnostics:\n\n```\n./ds4-bench \\\n  -m gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf \\\n  --prompt-file speed-bench/promessi_sposi.txt \\\n  --ctx-start 32768 \\\n  --ctx-max 65536 \\\n  --step-incr 32768 \\\n  --gen-tokens 0 \\\n  --role coordinator \\\n  --layers 0:19 \\\n  --listen 169.254.43.68 1234 \\\n  --debug\n```\n\n`--debug`\n\non the coordinator prints route formation and per-hop telemetry:\nlayer range, token span, local evaluation time, downstream wait time, socket\nsend time, and input/output byte counts. This is the current profiling tool for\ndeciding whether a split is balanced. `--dist-prefill-window N`\n\ncontrols how\nmany prefill chunks may be in flight end-to-end; the default is conservative\nand bounded. `--dist-prefill-chunk N`\n\nexists for experiments, but the default\n4096-token chunk is the canonical setting and should be used unless you are\nexplicitly validating a different chunk size.\n\nBy default DwarfStar sends hidden-state activations as 32-bit floats. To reduce\ntraffic, pass `--dist-activation-bits 16`\n\nor `--dist-activation-bits 8`\n\non the\ncoordinator. This changes only the transport format between machines, not the\nmodel weights or KV cache. 16-bit transport halves activation traffic and is the\nfirst option to try on Ethernet or WiFi. 8-bit transport is more aggressive and\nshould be treated as an approximate/experimental mode unless you have validated\nthe output for your use case. However experimentally reduction activation\nsize didn't provide a significant improvement, so this option may be removed\nin the future.\n\n**If a worker disconnects, the coordinator removes that worker from the active\nroute**. The request already in flight can fail, and later calls report an\nincomplete route until a compatible worker reconnects and sends a new\nregistration. For live sessions, the coordinator keeps the token history and can\nrebuild worker KV state by replaying the prefix when the route is available\nagain. Workers also validate a rolling 64-bit token-prefix hash on every work\nitem, so a restarted worker at position 0 cannot silently accept work for\nposition N; it reports the mismatch and the coordinator replays the current\ntranscript. Ctrl+C in the CLI and agent is cooperative: DwarfStar waits for the\ncurrent distributed token or prefill chunk to drain before returning control,\nwhich avoids coordinator-caused KV splits. Saved agent/server sessions use the\nsame KV file format as single-machine sessions: during save the coordinator\nfetches worker-owned layer tensors and serializes one normal payload; during\nload it splits that payload over the currently registered route.\n\nAt the protocol level there are two kinds of connections. Workers keep a\ncontrol TCP connection open to the coordinator and send a `HELLO`\n\nwith their\nmodel ID, model family, quant profile, layer slice, context capacity, and data\nport. The coordinator uses these registrations to build a route that covers all\nlayers. Work then moves over low-latency TCP data connections: the coordinator\ncomputes the first slice, sends a `WORK`\n\nframe with session ID, token positions,\nrolling token-prefix hashes before and after the span, route information, and\nhidden-state payload, and each worker computes its slice. Middle workers can\nforward directly to the next worker. The final worker returns logits to the\ncoordinator, or ACKs for non-final prefill chunks so the prefill pipeline can\nstay full. `RESULT`\n\nframes echo the request ID and the post-span hash. A worker\nstatus error is handled differently from a socket failure: KV/hash mismatch can\nbe recovered by replaying the token history on the same route, while transport\nfailure drops the route and waits for a replacement worker. For persistent KV,\nthe coordinator opens worker data connections and sends snapshot save/load\nmessages for each worker-owned layer range; the disk payload remains a single\nagent/server cache file. The protocol has no\nencryption or authentication, and is not release-stable yet; coordinator and\nworkers should be built from the same commit and used on trusted machines and\ntrusted networks.\n\nTensor parallelism runs a single decode across two Macs connected with a\nThunderbolt 5 cable, splitting the heavy per-layer work between the two\nGPUs and exchanging 16-24KB partial sums at synchronization gates inside the\ngraph (RDMA over Thunderbolt when available, a dedicated TCP socket\notherwise). Unlike the pipelined distributed mode above, both\nmachines work on the *same token at the same time*, so it reduces\nper-token latency instead of just fitting a bigger model.\n\nEach machine keeps one contiguous half of the routed experts resident. Dense, attention, shared-expert, embedding, and output weights remain replicated. This lets a model whose routed experts do not fit on one machine run fully resident across the pair; routed kernels never touch the peer's expert half.\n\nOne-time setup per boot, on **both** machines:\n\n```\n# Let the GPU wire ~117 GB (default cap is ~75% of RAM; the resident\n# expert shard needs ~97.5 GiB plus KV/scratch).\nsudo sysctl iogpu.wired_limit_mb=120000\n\n# RDMA over Thunderbolt needs an IPv4 address directly on the cabled\n# member interface (the bridge IP does not count). Use the interface\n# that is 'active' in ifconfig, e.g. en1 on one side and en6 on the\n# other. Skip this if you are fine with the TCP fallback.\nsudo ifconfig en1 inet 10.99.0.2/30 alias     # machine A\nsudo ifconfig en6 inet 10.99.0.1/30 alias     # machine B\n```\n\nCheck the verbs device before loading the model:\n\n```\nrdma_ctl status\nibv_devinfo -v\n```\n\nThe device must be active and expose the IPv4-mapped GID for the address above,\nfor example `::ffff:10.99.0.2`\n\n. A working IP ping does not prove that RDMA is\nactive.\n\nBoth machines need the same tree, commit, and GGUF path. Tensor parallelism is\nalways a 50/50 split with one worker, so do not pass `--layers`\n\n. Start the worker\nfirst; it retries while the coordinator loads. The worker must dial the address\non the Thunderbolt member interface, not the bridge address:\n\n```\nMODEL=gguf/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf\n\n# Machine B: worker.\n./ds4 -m \"$MODEL\" --tensor-parallel --role worker \\\n  --coordinator 10.99.0.2 9911 --transport rdma\n\n# Machine A: coordinator.\n./ds4 -m \"$MODEL\" --tensor-parallel --role coordinator \\\n  --listen 10.99.0.2 9911 --transport rdma -c 8192 \\\n  -p \"Tell me something about the sea.\"\n```\n\nThe active verbs device and IPv4-mapped GID are selected automatically. If that\nis ambiguous, add `--rdma-device rdma_en6 --rdma-gid-index 1`\n\non the worker and\nthe matching `rdma_en1`\n\nflags on the coordinator. Use `--transport tcp`\n\non both\nsides to force TCP. Tensor parallel roles are currently exposed by the `ds4`\n\nCLI, not by `ds4-server`\n\nor `ds4-agent`\n\n.\n\nStartup takes about 9 seconds per machine: each rank pre-faults its ~100 GiB shard from SSD and pins it through a Metal residency set. DeepSeek V4 Flash works the same way with its own GGUF on both machines. DeepSeek gate vectors are 16 KB and ride as one RDMA message. GLM's 6144-wide 24 KB vectors are split into two ordered RDMA messages.\n\nMeasured on two M5 Max 128 GB MacBooks (GLM 5.2, IQ2_XXS, 188 GiB):\n\n| two Macs, tensor parallel | one Mac, SSD streaming | |\n|---|---|---|\n| decode | ~16.8 t/s (15.4 at 4k context) | ~4.8 t/s |\n| prefill (4096 tokens) | ~94 t/s | ~3-5 t/s |\n| residency | fully memory-resident | streams experts from SSD |\n\nNotes: the coordinator mirrors every prompt sync and eval to the worker, so\nboth KV caches stay in lockstep; prompt processing splits both the\nrouted-expert GEMMs (by expert ownership) and the attention heads (a\ncontiguous half per machine) with one bulk partial-sum exchange per\nlayer per stage (`--tensor-parallel-token-prefill`\n\nselects a slower\ntoken-by-token prefill that exactly matches the single-machine arithmetic).\nThe split graph is deterministic, but its changed floating-point reduction\norder is not generally byte-identical to single-machine execution.\n\nOn a single CUDA server, `--cuda-tensor-parallel`\n\nsplits DeepSeek V4 Flash\ntensor and routed-expert work across an even number of GPUs. This is separate\nfrom the Mac-to-Mac mode above: it does not use `--role`\n\n, RDMA, or the\ndistributed layer pipeline. GPU placement and memory budgets are selected with\nthe normal `--gpu-devices`\n\nand `--gpu-vram`\n\noptions.\n\nThe device order is significant. With `N`\n\ndevices, the first `N/2`\n\nlogical\ntiers are contiguous layer-pipeline homes and the second `N/2`\n\ntiers are their\ntensor-parallel partners. Specify all homes first and then all partners, with\nthe closest P2P pair at matching positions. For example, the tested L40S host\nuses physical pairs `(0,1)`\n\n, `(2,3)`\n\n, `(4,5)`\n\n, and `(6,7)`\n\n, expressed as\n`0,2,4,6,1,3,5,7`\n\n. Each pair stores a 50/50 split of the routed experts, and\nthe vocabulary head is row-sharded across the participating output tiers.\nThose large tensors are not duplicated. Dense attention, router, and shared\nexpert weights are replicated within each pair.\n\nFor maximum throughput on eight 48 GB L40S cards, use the imatrix Q4 model.\nIts routed `Q4_K`\n\nlayout has the native grouped multi-session kernels; the Q2\nmodel is the lower-memory choice (including tested four-card runs), but its\nunsupported grouped routed shapes use the exact fallback and have lower\naggregate serving throughput. Download and build the L40S target with:\n\n```\n./download_model.sh ds4f-q4\nmake cuda CUDA_ARCH=sm_89\n```\n\nThis is the interactive-agent setup used on the eight-L40S server:\n\n```\nMODEL=gguf/DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-0731.gguf\n\n./ds4-agent --cuda --cuda-tensor-parallel \\\n  --gpu-vram auto \\\n  --gpu-devices 0,2,4,6,1,3,5,7 \\\n  --model \"$MODEL\" \\\n  --ctx 100000\n```\n\nFor serving, keep multiple KV sessions resident so decode rows can be grouped across requests. The tested host is configured for up to 16 resident sessions:\n\n```\n./ds4-server --cuda --cuda-tensor-parallel \\\n  --gpu-vram auto \\\n  --gpu-devices 0,2,4,6,1,3,5,7 \\\n  --model \"$MODEL\" \\\n  --ctx 100000 \\\n  --batched-session 16 \\\n  --host 0.0.0.0\n```\n\nThe equivalent local launchers are `./run-nvidia-tp-agent.sh`\n\nand\n`./run-nvidia-tp-server.sh`\n\n. The server launcher also enables the on-disk KV\ncache and defaults to the native 0731 MXFP4 GGUF. Set `DS4_MODEL`\n\nto use the Q4\nfile above instead. Reduce the session count or context size if the requested\nresident KV caches do not fit after model loading. CUDA TP, half-resident expert\nownership, output sharding, pipelined prefill, and compatible grouped decode are\nselected by `--cuda-tensor-parallel`\n\n; no `DS4_CUDA_*`\n\nenvironment tuning is required.\nWithout an explicit `--prefill-chunk`\n\n, this mode uses 2048-token chunks so the\ntested 16-session, 100k-context layout retains enough VRAM for resident KV\ncaches. An explicit `--prefill-chunk`\n\nremains an override for other topologies.\n\nAny even card count that can hold the selected model and graph scratch is a\nvalid topology. On this class of 48 GB card, the useful measured endpoints are\nQ2 on four cards (two pipeline stages) and Q4 on eight cards (four stages).\nFor a four-card PIX-paired subset such as physical GPUs `0,1,4,5`\n\n, the ordered\nlist is `0,4,1,5`\n\n. Two cards do not have enough memory for these Flash models.\n\nThis mode currently requires DeepSeek V4 Flash and an even multi-GPU\nplacement. GLM 5.2 instead uses normal layer placement across the selected\nCUDA devices. DGX Spark is a single-GPU target and must not be started with\n`--cuda-tensor-parallel`\n\n.\n\nLong local inference runs can keep the GPU busy for extended periods. If you\ncare more about heat, fan noise, battery life on MacBooks, or reducing thermal\nstress on the hardware than about maximum throughput, use `--power N`\n\n.\n\n`--power 100`\n\nis the default and means full speed. Lower values ask DwarfStar to target\nthat percentage of GPU usage: `--power 70`\n\ntargets about 70%, `--power 50`\n\ntargets about half usage, and so forth. DwarfStar does this by measuring GPU work time\nand inserting small sleeps between work units: during prefill it sleeps between\nlayers, and during generation it sleeps between decoded tokens. This reduces\nsustained load without changing model output.\n\nThe option is available on the CLI, server, agent, eval, and benchmark tools\nfor DeepSeek models. GLM 5.2 currently accepts only `--power 100`\n\n. For example:\n\n```\n./ds4 --power 50\n./ds4-agent --power 70\n./ds4-server --power 40 --ctx 100000\n```\n\nDwarfStar features a native coding agent that works in a different way than most other systems: the inference is controlled from within the agent itself, without socket/API boundaries, so the session is represented by the on-disk KV cache itself. Moreover the tools and the system prompt are all designed vertically for DeepSeek v4 Flash and PRO. This provides a few advantages:\n\n- Low latency experience, bounded mainly by the prefill speed limits. Displaying of generated text, tool calling, start of a new session are always instantaneous.\n- Live progress bar during prefill time.\n- No DSML tool calling conversion, the tools are handled natively in the LLM format.\n- KV cache mismatch are impossible by construction, the current state is always the truth.\n- Everything is tuned for this model.\n- Ability to switch saved sessions with\n`/list`\n\nand`/switch`\n\n; full KV sessions resume without a prefill stage.\n\nAgent sessions are stored in `~/.ds4/kvcache`\n\n. Use `/save`\n\nto persist the\ncurrent session, `/list`\n\nto show saved sessions sorted by recent update time,\nand `/switch <sha>`\n\nto resume one of them. The session ID is stable across\nfuture saves and is derived from the first user prompt and creation time.\n`/del <sha>`\n\nremoves a saved session. `/strip <sha>`\n\nkeeps the rendered\nconversation text and title but removes the heavy KV payload; switching to a\nstripped session rebuilds the KV cache by prefilling the saved text.\n\nUse `--chdir /path/to/ds4`\n\nwhen launching `ds4-agent`\n\nfrom another directory,\nso relative runtime files such as `metal/*.metal`\n\nresolve from the project tree.\n\nHowever while the system already works, there is a lot of work to do\nin order to make it ready for prime time. When finally the agent will reach\nthe wanted shape, we will *likely* split the server and the client creating a stateful\nsession-based protocol that can recreate all that in a client-server way.\n\n`ds4-bench`\n\nmeasures instantaneous prefill and generation throughput at context\nfrontiers instead of reporting one whole-run average. It loads the model once,\nwalks a fixed token sequence to frontiers such as 2048, 4096, 6144, and uses\nincremental prefill so each row measures only the newly-added token interval.\nAfter each frontier it saves the live KV state to memory, generates a fixed\ngreedy non-EOS probe, restores the memory snapshot, and continues prefill.\n\n```\n./ds4-bench \\\n  -m ds4flash.gguf \\\n  --prompt-file speed-bench/promessi_sposi.txt \\\n  --ctx-start 2048 \\\n  --ctx-max 65536 \\\n  --step-incr 2048 \\\n  --gen-tokens 128\n```\n\nThe example file is a cleaned public-domain Project Gutenberg text of\nAlessandro Manzoni's *I Promessi Sposi* (ebook #45334), with the Gutenberg\nheader and footer removed: [https://www.gutenberg.org/ebooks/45334](https://www.gutenberg.org/ebooks/45334).\n\nUse `--step-incr N`\n\nfor different linear spacing, or `--step-mul F`\n\nfor\nexponential sweeps. Output is CSV with one row per frontier: latest prefill\ninterval tokens/sec, generation tokens/sec at that frontier, and\n`kvcache_bytes`\n\n.\n\nSessions prefill long prompts in 4096-token chunks by default. Use\n`--prefill-chunk 2048`\n\n, for example, to match the strict official-vector\ncheckpoint path. Changing the chunk changes the KV checkpoint/logit path, so\ncompare it as an explicit run configuration.\nChunked Metal prefill reuses the same range-capable layer-major graph for each\nchunk, preserving absolute compressor/indexer boundaries while avoiding the old\nper-layer chunk dispatch path.\n\n`ds4-eval`\n\nis a small real-model integration benchmark. It is not a leaderboard\nrunner and should not be reported as an official GPQA, SuperGPQA, AIME, or\nsecurity benchmark score: the questions are an embedded 92-item subset chosen\nto make local regression testing useful and visually inspectable. The program\nloads the real GGUF, renders DeepSeek chat prompts, streams sampled tokens in a split-screen TUI, grades\nthe final answer, and prints a per-question report with prompt tokens,\ngenerated tokens, pass/fail state, the model answer, and the correct answer.\n\n```\n./ds4-eval -m ds4flash.gguf --trace /tmp/ds4-eval.txt\n```\n\nThe default run uses `--tokens 16000`\n\n, thinking mode enabled, and a soft/hard\n`</think>`\n\nbudget cutoff so the model has room to produce a visible answer.\n`ds4-eval`\n\nsizes the context internally from the largest selected prompt plus\nthe generation budget, and refuses runs that would need more than 1M context\ntokens. Press `p`\n\nto pause, `q`\n\nto exit and print the report, Up/Down to\ninspect or select another question, and Enter to run the selected question next.\n`--plain`\n\ndisables the TUI.\n\nUse `--regrade-trace /path/to/trace.txt`\n\nto replay the current answer\nextractor and scorer against a prior `--trace`\n\nfile without loading the model\nor regenerating tokens. This is useful when auditing evaluator changes: it\nshows which cases changed, the old picked answer, the new picked answer, and a\npass/fail summary.\n\nFor inference changes that can affect generation drift, keep this deterministic q1..q4 token-count gate in the test plan:\n\n```\n./ds4-eval \\\n  -m ds4flash.gguf \\\n  --plain \\\n  --questions 4 \\\n  --tokens 2048 \\\n  --temp 0 \\\n  --seed 1\n```\n\nThe generated-token counts must stay aligned with the baseline:\n\n| Question | Expected state | Expected generated tokens | Expected given/correct |\n|---|---|---|---|\n| 1 | `PASSED` |\n2048 | `B` / `B` |\n| 2 | `PASSED` |\n438 | `C` / `C` |\n| 3 | `PASSED` |\n666 | `70` / `70` |\n| 4 | `FAILED` |\n2048 | `A` / `C` |\n\nThe first 75 embedded questions are interleaved as 25 GPQA Diamond, 25 audited SuperGPQA, and 25 AIME 2025 problems. The final 17 are an audited COMPSEC subset of reduced single-function C/C++ vulnerability-localization questions. The model is asked for the single best source line, or the smallest exact line set only when the bug cannot be localized to one line; the scorer accepts small audited ranges only when adjacent lines are equivalent locations for the same bug. The order is intentionally progressive: early questions are useful smoke tests, while later questions are hard enough that a strong reasoning model should still miss some of them. The SuperGPQA slice is curated rather than blind: upstream rows with wrong keys, missing figures, or underspecified prompts are replaced with cleaner rows.\n\nThe set should be treated as a hard capability regression suite rather than a pass/fail unit test.\n\n**GPQA Diamond** contributes graduate-level science questions with multiple-choice answers. DeepSeek's model card reports strong results on full GPQA Diamond in thinking mode, but individual items still require careful physics, chemistry, or biology reasoning and are easy to lose with a small prompt/rendering or sampling regression.**SuperGPQA** contributes broad specialist knowledge and domain-transfer questions. The model-card SuperGPQA number is much lower than GPQA Diamond, so these items are expected to be uneven: some look mundane, others require niche professional knowledge or exact interpretation of a translated-style exam question.**AIME 2025** contributes exact-answer contest math. These are often the most unforgiving items in the set: no multiple-choice prior, no partial credit, and a single arithmetic or algebraic slip changes the grade.**COMPSEC** contributes single-function C/C++ security reasoning items reduced from public CVE writeups. These are not exploit prompts: the task is to identify the best source line where the defensive code flaw is introduced, or return`0`\n\nfor a safe function.\n\nIn practice this means `ds4-eval`\n\nshould not be expected to produce a perfect\n92/92 run. It is meant to answer a more useful engineering question: after a\nkernel, quantization, prompt-rendering, KV-cache, or tool-streaming change, does\nDeepSeek V4 Flash still solve a representative mix of hard science, broad\nknowledge, exact math, and security-code problems while using the same inference\npath users run?\n\nOne-shot prompt:\n\n```\n./ds4 -p \"Explain Redis streams in one paragraph.\"\n```\n\nNo `-p`\n\nstarts the interactive prompt:\n\n```\n./ds4\nds4>\n```\n\nThe interactive CLI is a real multi-turn chat. It keeps the rendered chat\ntranscript and the live graph KV checkpoint, so each turn extends the previous\nconversation. Useful commands are `/help`\n\n, `/think`\n\n, `/think-max`\n\n, `/nothink`\n\n,\n`/ctx N`\n\n, `/read FILE`\n\n, and `/quit`\n\n. Ctrl+C interrupts the current generation\nand returns to `ds4>`\n\n.\n\nThe CLI defaults to thinking mode. Use `/nothink`\n\nor `--nothink`\n\nfor direct\nanswers. `--mtp MTP.gguf --mtp-draft 2`\n\nenables the optional MTP speculative\npath; it is useful only for greedy decoding, currently uses a confidence gate\n(`--mtp-margin`\n\n) to avoid slow partial accepts, and should be treated as an\nexperimental slight-speedup path.\n\nStart a local OpenAI/Anthropic-compatible server:\n\n```\n./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192\n```\n\nUse `--chdir /path/to/ds4`\n\nwhen launching `ds4-server`\n\nfrom another directory,\nso relative runtime files such as `metal/*.metal`\n\nresolve from the project tree.\n\nBy default the server keeps one mutable backend/KV checkpoint in memory, so stateless clients that resend a longer version of the same prompt can reuse the shared prefix instead of pre-filling from token zero.\n\n`--batched-session N`\n\npreallocates `N`\n\nindependent resident KV sessions. Ready\ndecode steps are evaluated together, while long prefills alternate in bounded\nchunks so one request does not block every decoder. Requests beyond `N`\n\nwait\nfor a resident slot. If disk KV caching is enabled, an idle slot is persisted\nbefore reuse and can be restored when that conversation returns; an active\nrequest is never evicted. Choose `N`\n\nand `--ctx`\n\nso all resident KV allocations\nfit in GPU memory. Without this option, inference retains the original\nsingle-session behavior.\n\nWhile generation is active, prefill yields every 128 tokens by default.\n`--mixed-prefill-quantum N`\n\nchanges that interval for testing; larger values\nreduce scheduling handoffs but can make active decoders wait longer.\n\nDecode batching is exact: when a native batched kernel is unavailable, DwarfStar runs the affected rows in a fixed order and returns the same full logits as separate session evaluations. The current backend behavior is:\n\n| Backend and model | Session execution |\n|---|---|\n| Metal, resident DeepSeek Flash | Native shared-expert and QKV batching from two rows upward when supported; ordered fallback otherwise. |\n| Metal, GLM 5.2 | Ordered exact fallback. |\n| CUDA, DeepSeek Flash on a supported multi-GPU TP/EP layout | Native decode and mixed prefill/decode, with exact fallbacks for unsupported kernel shapes. |\n| CUDA single GPU, including DGX Spark | Ordered exact fallback. |\n\n`N`\n\nresident sessions allocate `N`\n\nKV states, so a context size that fits once\nmay not fit eight times. Native batching can improve aggregate throughput; an\nordered fallback provides concurrency and fairness, but not the same speedup.\nMTP speculative decoding is disabled while native session batching is active.\n\nSupported endpoints:\n\n`GET /v1/models`\n\n`GET /v1/models/deepseek-v4-flash`\n\n`GET /v1/models/deepseek-v4-pro`\n\n`POST /v1/chat/completions`\n\n`POST /v1/responses`\n\n`POST /v1/completions`\n\n`POST /v1/messages`\n\nThe Flash and PRO model endpoints are compatibility aliases. They both report\nthe model currently loaded from the GGUF passed with `-m`\n\n; the endpoint name does\nnot select a different model.\n\n`/v1/chat/completions`\n\naccepts the usual OpenAI-style `messages`\n\n,\n`max_tokens`\n\n/`max_completion_tokens`\n\n, `temperature`\n\n, `top_p`\n\n, `top_k`\n\n, `min_p`\n\n,\n`seed`\n\n, `stream`\n\n, `stream_options.include_usage`\n\n, `tools`\n\n, and `tool_choice`\n\n.\nTool schemas are rendered into DeepSeek's DSML tool format, and generated DSML\ntool calls are mapped back to OpenAI tool calls.\n\n`/v1/responses`\n\naccepts OpenAI Responses-style `input`\n\n, `instructions`\n\n,\n`tools`\n\n, `tool_choice`\n\n, `max_output_tokens`\n\n, `temperature`\n\n, `top_p`\n\n, `stream`\n\n,\nand `reasoning`\n\n. It is the preferred endpoint for Codex CLI. The server keeps\nResponses continuations bound to live state when possible, and can fall back to\nthe same DSML rendering and KV prefix reuse used by chat completions.\n\n`/v1/messages`\n\nis the Anthropic-compatible endpoint used by Claude Code style\nclients. It accepts `system`\n\n, `messages`\n\n, `tools`\n\n, `tool_choice`\n\n, `max_tokens`\n\n,\n`temperature`\n\n, `top_p`\n\n, `top_k`\n\n, `stream`\n\n, `stop_sequences`\n\n, and thinking\ncontrols. Tool uses are returned as Anthropic `tool_use`\n\nblocks.\n\nDefault sampled API generation uses `temperature=1`\n\n, `top_p=1`\n\n, and\n`min_p=0.05`\n\n, so the default filter is relative probability rather than\nnucleus mass. In thinking mode DwarfStar applies those fixed sampling defaults\nto any knob the request omits, matching DeepSeek's fixed-thinking API behavior,\nbut sampling parameters set explicitly in the request always win: a\n`temperature=0`\n\nrequest is greedy through the whole reasoning phase, so\nbenchmark harnesses get deterministic thinking-mode output.\n\nThe chat, Responses, and Anthropic endpoints support SSE streaming. In thinking\nmode, reasoning is streamed in the native API shape instead of being mixed into\nfinal text. OpenAI chat streaming\nalso streams tool calls as soon as the DSML invocation is recognized: the tool\nheader is sent first, then parameter bytes are forwarded as\n`tool_calls[].function.arguments`\n\ndeltas while generation continues. The\nAnthropic endpoint streams thinking and text live, then emits structured\n`tool_use`\n\nblocks when the generated tool block is complete.\nThe Responses endpoint streams the Responses event lifecycle expected by Codex,\nincluding `response.output_text.delta`\n\n, function-call argument events, and\nterminal `response.completed`\n\n/ `response.incomplete`\n\n/ `response.failed`\n\nevents.\n\nFor browser JavaScript clients served from another origin, start the server with\n`--cors`\n\nto emit `Access-Control-Allow-*`\n\nheaders. This only changes HTTP\nheaders; it does not expose the server on the LAN. Use `--host 0.0.0.0`\n\nexplicitly when remote machines should be able to connect.\n\nDeepSeek V4 emits tool calls as [DSML text](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/encoding/README.md). Agent clients do not send that\nsame text back on the next request: they send normalized OpenAI/Anthropic JSON\ntool-call objects. **If the server re-rendered those objects slightly\ndifferently, the rendered byte prefix would no longer match the live KV\ncheckpoint** and the next turn would have to be rebuilt.\n\nThe first line of defense is exact replay. Every tool call gets an unguessable\nAPI tool ID, and the server remembers `tool id -> exact sampled DSML block`\n\nin\na bounded in-memory map backed by radix trees. When the client later sends that\ntool ID back, the prompt renderer uses the exact DSML bytes the model sampled,\nnot a freshly formatted approximation. This map can also be saved inside KV\ncache files, so exact replay survives server restarts for cached histories.\n\n**Canonicalization is only the backup path**. If the exact DSML block is missing,\nor exact replay is disabled with `--disable-exact-dsml-tool-replay`\n\n, the server\nrenders a deterministic DSML form from the JSON tool object. After a tool-call\nturn, it compares the live sampled token stream with the prompt that the next\nclient request will render. If needed, it rewrites the live checkpoint, or\nfalls back to an older disk KV snapshot and replays only the suffix. This keeps\nthe model continuation aligned with the stateless API transcript.\n\nDuring generation, the server also treats DSML syntax differently from payload.\nWhen the model is emitting stable protocol structure such as DSML tags,\nparameter headers, JSON punctuation, or closing markers, sampling is forced to\n`temperature=0`\n\nso the tool call stays parseable. This greedy mode does **not**\napply to argument payloads: `string=true`\n\nparameter bodies and JSON string\nvalues, including file contents and edit text, use the request's normal sampling\nsettings. That separation is important: deterministic decoding is helpful for\nsyntax, but can create repeated text when applied to long code or file bodies.\n\nMinimal OpenAI example:\n\n```\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"model\":\"deepseek-v4-flash\",\n    \"messages\":[{\"role\":\"user\",\"content\":\"List three Redis design principles.\"}],\n    \"stream\":true\n  }'\n```\n\n`ds4-server`\n\ncan be used by local coding agents that speak OpenAI-compatible\nchat completions. Start the server first, and set the client context limit no\nhigher than the `--ctx`\n\nvalue you started the server with:\n\n```\n./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192\n```\n\nYou can use larger context and larger cache if you wish. Full context of 1M tokens is going to use more or less 26GB of memory (compressed indexer alone will be like 22GB), so configure a context which makes sense in your system. With 128GB of RAM you would run the 2-bit quants, which are already 81GB, 26GB are going to be likely too much, so a context window of 100~300k tokens is wiser. However users reported being able to run 2bit quants with 250k ctx window in a Macs with just 96GB of system memory: make sure to kill processes that use too much memory, if you plan doing so ;)\n\nThe `384000`\n\noutput limit below avoids token caps since the model is able\nto generate very long replies otherwise (up to 384k tokens). The server\nstill stops when the configured context window is full.\n\nFor **opencode**, add a provider and agent entry to\n`~/.config/opencode/opencode.json`\n\n:\n\n```\n{\n  \"$schema\": \"https://opencode.ai/config.json\",\n  \"provider\": {\n    \"ds4\": {\n      \"name\": \"ds4.c (local)\",\n      \"npm\": \"@ai-sdk/openai-compatible\",\n      \"options\": {\n        \"baseURL\": \"http://127.0.0.1:8000/v1\",\n        \"apiKey\": \"dsv4-local\"\n      },\n      \"models\": {\n        \"deepseek-v4-flash\": {\n          \"name\": \"DeepSeek V4 Flash (ds4.c local)\",\n          \"limit\": {\n            \"context\": 100000,\n            \"output\": 384000\n          }\n        }\n      }\n    }\n  },\n  \"agent\": {\n    \"ds4\": {\n      \"description\": \"DeepSeek V4 Flash served by local ds4-server\",\n      \"model\": \"ds4/deepseek-v4-flash\",\n      \"temperature\": 0\n    }\n  }\n}\n```\n\nFor **Pi**, add a provider to `~/.pi/agent/models.json`\n\n:\n\n```\n{\n  \"providers\": {\n    \"ds4\": {\n      \"name\": \"ds4.c local\",\n      \"baseUrl\": \"http://127.0.0.1:8000/v1\",\n      \"api\": \"openai-completions\",\n      \"apiKey\": \"dsv4-local\",\n      \"compat\": {\n        \"supportsStore\": false,\n        \"supportsDeveloperRole\": false,\n        \"supportsReasoningEffort\": true,\n        \"supportsUsageInStreaming\": true,\n        \"maxTokensField\": \"max_tokens\",\n        \"supportsStrictMode\": false,\n        \"thinkingFormat\": \"deepseek\",\n        \"requiresReasoningContentOnAssistantMessages\": true\n      },\n      \"models\": [\n        {\n          \"id\": \"deepseek-v4-flash\",\n          \"name\": \"DeepSeek V4 Flash (ds4.c local)\",\n          \"reasoning\": true,\n          \"thinkingLevelMap\": {\n            \"off\": null,\n            \"minimal\": \"low\",\n            \"low\": \"low\",\n            \"medium\": \"medium\",\n            \"high\": \"high\",\n            \"xhigh\": \"xhigh\"\n          },\n          \"input\": [\"text\"],\n          \"contextWindow\": 100000,\n          \"maxTokens\": 384000,\n          \"cost\": {\n            \"input\": 0,\n            \"output\": 0,\n            \"cacheRead\": 0,\n            \"cacheWrite\": 0\n          }\n        }\n      ]\n    }\n  }\n}\n```\n\nOptionally make it the default Pi model in `~/.pi/agent/settings.json`\n\n:\n\n```\n{\n  \"defaultProvider\": \"ds4\",\n  \"defaultModel\": \"deepseek-v4-flash\"\n}\n```\n\nFor **Codex CLI**, use the Responses wire API:\n\n```\n[model_providers.ds4]\nname = \"DS4\"\nbase_url = \"http://127.0.0.1:8000/v1\"\nwire_api = \"responses\"\nstream_idle_timeout_ms = 1000000\n```\n\nThen run:\n\n```\ncodex --model deepseek-v4-flash -c model_provider=ds4\n```\n\nFor **Claude Code**, use the Anthropic-compatible endpoint. A wrapper like this\nmatches the local `~/bin/claude-ds4`\n\nsetup:\n\n``` bash\n#!/bin/sh\nunset ANTHROPIC_API_KEY\n\nexport ANTHROPIC_BASE_URL=\"http://127.0.0.1:8000\"\nexport ANTHROPIC_AUTH_TOKEN=\"dsv4-local\"\nexport ANTHROPIC_MODEL=\"deepseek-v4-flash\"\n\nexport ANTHROPIC_CUSTOM_MODEL_OPTION=\"deepseek-v4-flash\"\nexport ANTHROPIC_CUSTOM_MODEL_OPTION_NAME=\"DeepSeek V4 Flash local ds4\"\nexport ANTHROPIC_CUSTOM_MODEL_OPTION_DESCRIPTION=\"ds4.c local GGUF\"\n\nexport ANTHROPIC_DEFAULT_SONNET_MODEL=\"deepseek-v4-flash\"\nexport ANTHROPIC_DEFAULT_HAIKU_MODEL=\"deepseek-v4-flash\"\nexport ANTHROPIC_DEFAULT_OPUS_MODEL=\"deepseek-v4-flash\"\nexport CLAUDE_CODE_SUBAGENT_MODEL=\"deepseek-v4-flash\"\n\nexport CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1\nexport CLAUDE_CODE_DISABLE_NONSTREAMING_FALLBACK=1\nexport CLAUDE_STREAM_IDLE_TIMEOUT_MS=600000\n\nexec \"$HOME/.local/bin/claude\" \"$@\"\n```\n\nClaude Code may send a large initial prompt, often around 25k tokens, before it\nstarts doing useful work. Keep `--kv-disk-dir`\n\nenabled: after the first expensive\nprefill, the disk KV cache lets later continuations or restarted sessions reuse\nthe saved prefix instead of processing the whole prompt again.\n\nDeepSeek V4 Flash has distinct non-thinking, thinking, and Think Max modes.\nThe server defaults to thinking mode. `reasoning_effort=max`\n\nrequests Think\nMax, but it is only applied when the context size is large enough for the model\ncard recommendation; smaller contexts fall back to normal thinking. OpenAI\n`reasoning_effort=xhigh`\n\nstill maps to normal thinking, not Think Max.\n\nFor direct replies, use `thinking: {\"type\":\"disabled\"}`\n\n, `think:false`\n\n, or a\nnon-thinking model alias such as `deepseek-chat`\n\n.\n\nChat/completion APIs are stateless: agent clients usually resend the whole\nconversation every request. `ds4-server`\n\nfirst tries the cheap exact token-prefix\ncheck, then falls back to comparing rendered prompt bytes with decoded\ncheckpoint bytes. The live in-memory checkpoint covers the current session; the\ndisk KV cache makes useful prefixes survive session switches and server\nrestarts.\n\nFor RAM reasons there is currently only one live KV cache in memory. When a new unrelated session replaces it, the old checkpoint can only be resumed without re-processing if it was written to the disk KV cache. In other words, memory cache handles the active session; disk cache is the resume mechanism for different sessions.\n\nEnable it with:\n\n```\n./ds4-server --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192\n```\n\nThe cache key is the SHA1 of the rendered byte prefix, and files are named\n`<sha1>.kv`\n\n. The DS4 payload still stores the exact token IDs and graph state\nfor that prefix. This matters for continued chats: the model may have generated\none token whose decoded text is later sent back by a client as two canonical\nprompt tokens. A rendered byte-prefix hit can still reuse the checkpoint and\ntokenize only the new suffix.\nThe file is intentionally written with ordinary `read`\n\n/`write`\n\nI/O, not\n`mmap`\n\n, so restoring cache entries does not add more VM mappings to a process\nthat already maps the model.\n\nTool calls also keep a bounded exact-DSML replay map keyed by unguessable tool\nIDs, so client JSON history can be rendered back to the exact sampled text. The\nRAM map keeps up to 100000 IDs by default; tune it with `--tool-memory-max-ids`\n\n.\nUse `--disable-exact-dsml-tool-replay`\n\nto disable this and fall back to\ncanonical JSON-to-DSML rendering.\n\nOn disk, a cache file is:\n\n```\nKVC fixed header, 48 bytes\nu32 rendered_text_bytes\nrendered_text_bytes of UTF-8-ish token text\nDS4 session payload, payload_bytes from the KVC header\noptional tool-id map section\n```\n\nThe fixed header is little-endian:\n\n```\n0   u8[3]  magic = \"KVC\"\n3   u8     version = 1\n4   u8     routed expert quant bits, currently 2 or 4\n5   u8     save reason: 0 unknown, 1 cold, 2 continued, 3 evict, 4 shutdown\n6   u8     extension flags, bit 0 = appended tool-id map\n7   u8     reserved\n8   u32    cached token count\n12  u32    hit count\n16  u32    context size the snapshot was written for\n20  u8[4]  reserved\n24  u64    creation Unix time\n32  u64    last-used Unix time\n40  u64    DS4 session payload byte count\n```\n\nThe rendered text is the tokenizer-decoded text for the cached token prefix. It is both the human-inspectable prefix and the lookup identity: its SHA1 is the filename, and a file is reusable only when those bytes are a prefix of the incoming rendered prompt. After load, the exact checkpoint tokens from the DS4 payload remain authoritative, and only the incoming text suffix after the cached bytes is tokenized.\n\nThe optional tool-id map is present only when header extension bit 0 is set. Appended sections use fixed bit order, so future extension bits can add fields without ambiguity. The map stores unguessable API tool call IDs back to the exact DSML block the model sampled. Only mappings whose DSML block is present in the rendered cached text are stored. This lets restarted servers render later client history byte-for-byte like the original model output, even if the client reorders JSON arguments.\n\nThe current tool-id map section is:\n\n```\n0   u8[3]  magic = \"KTM\"\n3   u8     version = 1\n4   u32    entry count\n\nFor each entry:\n0   u32    tool id byte length\n4   u32    sampled DSML byte length\n8   bytes  tool id\n... bytes  exact sampled DSML block\n```\n\nThe section is auxiliary replay memory, not model state. A cache hit restores the session payload first, then loads the map if present. Before rendering a request, the server can also scan cache files for the tool IDs present in the client history and load just those mappings, so an exact DSML replay can survive server restarts even when the matching KV snapshot is not the one ultimately used for the rendered-prefix hit.\n\nThe DS4 session payload starts with thirteen little-endian `u32`\n\nfields:\n\n```\n0   magic = \"DSV4\"\n1   payload version = 2\n2   saved context size\n3   prefill chunk size\n4   raw KV ring capacity\n5   raw sliding-window length\n6   compressed KV capacity\n7   checkpoint token count\n8   layer count\n9   raw/head KV dimension\n10  indexer head dimension\n11  vocabulary size\n12  live raw rows serialized below\n```\n\nThen it stores:\n\n`u32[token_count]`\n\ncheckpoint token IDs.`float32[vocab_size]`\n\nlogits for the next token after that checkpoint.`u32[layer_count]`\n\ncompressed attention row counts.`u32[layer_count]`\n\nratio-4 indexer row counts.- For every layer: the live raw sliding-window KV rows, written in logical position order rather than physical ring order.\n- For compressed layers: live compressed KV rows and compressor frontier tensors.\n- For ratio-4 compressed layers: live indexer compressed rows and indexer frontier tensors.\n\nThe logits are raw IEEE-754 `float32`\n\nvalues from the host `ds4_session`\n\nbuffer. They are saved immediately after the checkpoint tokens so a loaded\nsnapshot can sample or continue from the exact next-token distribution without\nrunning one extra decode step. MTP draft logits/state are not persisted; after\nloading a disk checkpoint the draft state is invalidated and rebuilt by normal\ngeneration.\n\nDistributed coordinator sessions use the same `DSV4`\n\npayload. Worker-owned\nlayer tensors are pulled during save and merged into the normal layer-ordered\ntensor stream; during load the coordinator splits that stream into the current\nroute and pushes the relevant layer tensors back to the workers. The saved file\ndoes not retain the distributed topology.\n\nThe tensor payload is DS4-specific KV/session state, not a generic inference\ngraph dump. It is expected to be portable only across compatible `ds4.c`\n\nbuilds for this model layout.\n\nThe cache stores checkpoints at four moments:\n\n`cold`\n\n: after a long first prompt reaches a stable prefix, before generation.`continued`\n\n: when prefill or generation reaches the next absolute aligned frontier.`evict`\n\n: before an unrelated request replaces the live in-memory session.`shutdown`\n\n: when the server exits cleanly.\n\nCold saves intentionally trim a small token suffix and align down to a prefill chunk boundary. This avoids common BPE boundary retokenization misses when a future request appends text to the same prompt. The defaults are conservative: store prefixes of at least 512 tokens, cold-save prompts up to 30000 tokens, trim 32 tail tokens, and align to 2048-token chunks. The important knobs are:\n\nContinued saves use the same alignment and are written only when the live graph naturally reaches an absolute frontier. With the defaults this means roughly every 10k tokens, independent of where the first cold checkpoint landed, so long generations leave restart points behind without persisting the fragile final few tokens.\n\n`--kv-cache-min-tokens`\n\n`--kv-cache-cold-max-tokens`\n\n`--kv-cache-continued-interval-tokens`\n\n`--kv-cache-boundary-trim-tokens`\n\n`--kv-cache-boundary-align-tokens`\n\n`--tool-memory-max-ids`\n\n`--disable-exact-dsml-tool-replay`\n\nBy default, checkpoints may be reused across the 2-bit and 4-bit routed-expert\nvariants if the rendered prefix matches. Use `--kv-cache-reject-different-quant`\n\nwhen you want strict same-quant reuse only.\n\nThe cache directory is disposable. If behavior looks suspicious, stop the server and remove it. You can investigate what is cached with hexdump as the kv cache files include the verbatim prompt cached.\n\nThe default graph backend is Metal on macOS and CUDA in CUDA builds:\n\n```\n./ds4 -p \"Hello\" --metal\n./ds4 -p \"Hello\" --cuda\n```\n\nOn Linux, plain `make`\n\nprints the available build targets instead of selecting a\nCUDA target implicitly. Use `make cuda-spark`\n\nfor DGX Spark / GB10. It omits an\nexplicit `nvcc -arch`\n\nbecause that is currently the fastest path on GB10. Use\n`make cuda-generic`\n\nfor a normal local CUDA build, or set `CUDA_ARCH`\n\nexplicitly\nwhen cross-building or when you need a known target:\n\n```\nmake cuda CUDA_ARCH=sm_120\nmake cuda CUDA_ARCH=native\n```\n\nCUDA builds accept `--gpu-vram N[,N,...]`\n\nand `--gpu-devices N[,N,...]`\n\nin the\nCLI, server, agent, and benchmark. VRAM values are per-device GiB budgets;\n`--gpu-vram auto`\n\nuses the free memory reported by CUDA. The device list controls\nthe placement order and must have the same number of entries as an explicit\nbudget list. Placement reserves graph and KV memory for the requested context\nand refuses to start if model layers would spill to the CPU.\n\nWithout `--cuda-tensor-parallel`\n\n, CUDA uses normal layer placement across the\nlisted devices. This is also the supported multi-GPU layout for GLM 5.2:\n\n```\n./ds4 -m gguf/GLM-5.2-UD-Q2_K_RoutedQ2K.gguf \\\n  --gpu-vram auto --gpu-devices 0,2,4,6,1,3,5,7 \\\n  --ctx 32768 -p \"Hello\"\n```\n\nFor the DeepSeek Flash tensor/expert-parallel layout, see \"Tensor Parallelism across CUDA GPUs\" above.\n\nThere is also a CPU reference/debug path:\n\n```\n./ds4 -p \"Hello\" --cpu\nmake cpu\n./ds4\n./ds4 -p \"Hello\"\n```\n\nDo not treat the CPU path as the production target. The CLI and `ds4-server`\n\nsupport the CPU backend for reference/debug use and share the same KV session\nand snapshot format as Metal and CUDA, but normal inference should use Metal or\nCUDA.\n\nThis project supports steering with single-vector activation directions; see the\n`dir-steering`\n\ndirectory for more information. This follows the core idea of the\n[Refusal in Language Models Is Mediated by a Single Direction](https://arxiv.org/abs/2406.11717)\npaper. You can use it to make the model more or less verbose, less likely to\nanswer programming questions if it is a chatbot for your car rental web site,\nand so forth, much faster than fine-tuning.\nThis is also useful for cybersecurity researchers who want to reduce a model's\nwillingness to provide dual-use or offensive security guidance.\n\n`tests/test-vectors`\n\ncontains short and long-context continuation vectors\ncaptured from the official DeepSeek V4 Flash API. The requests use\n`deepseek-v4-flash`\n\n, greedy decoding, thinking disabled, and the maximum\n`top_logprobs`\n\nslice exposed by the API. Local vectors are generated with\n`./ds4 --dump-logprobs`\n\nand compared by token bytes, so tokenizer/template or\nattention regressions show up before they become long generation failures. The\nC runner pins a 2048-token prefill chunk for this strict API-vector comparison.\n\nThe core local tests are driven by the C runner, with a small `ds4-eval`\n\nextractor self-test run first:\n\n```\nmake test                  # ./ds4-eval --self-test-extractors && ./ds4_test --all\n./ds4_test --logprob-vectors\n./ds4_test --server\n```\n\nThe batching tests are model-backed and must run on the matching GPU backend:\n\n```\n# Metal, with DS4_TEST_SESSION_COUNT set to 2, 4, 8, and 16.\nDS4_TEST_MODEL=/path/to/model.gguf DS4_TEST_SESSION_COUNT=4 \\\n  make test-metal-session-batch\n\n# CUDA multi-GPU Flash.\nDS4_TEST_MODEL=/path/to/model.gguf make test-cuda-session-batch\nDS4_TEST_MODEL=/path/to/model.gguf make test-cuda-mixed-batch\n```\n\nFor GLM, run the same Metal session test with a GLM GGUF and run\n`tests/glm_long_context_smoke.sh /path/to/model.gguf`\n\n. The official 100-case\nquality scorers, two-Mac TCP/RDMA tests, CUDA matrix, and manual agent checks are\nrelease gates rather than quick local tests; follow\n[QA_BEFORE_RELEASES.md](/baker27727/ds4-8gb-cpu/blob/main/QA_BEFORE_RELEASES.md).\n\nWhen a generation looks wrong, three small tools are usually enough to get a first answer:\n\n```\n./ds4 --dump-tokens -p \"...\"\n./ds4 --dump-logprobs /tmp/out.json --logprobs-top-k 20 --temp 0 -p \"...\"\n./ds4 --dump-logits /tmp/logits.json --metal --nothink --prompt-file prompt.txt\n./ds4-server --trace /tmp/ds4-trace.txt ...\n```\n\n`--dump-tokens`\n\ntokenizes the`-p`\n\nor`--prompt-file`\n\nstring exactly as written, recognizes DS4 protocol specials, and then exits before inference starts. For example, the DSML tool close marker starts as two tokens:`</`\n\nand`｜DSML｜`\n\n.`--dump-logprobs`\n\nstores a greedy continuation with the top local alternatives at each step, which helps separate sampling choices from logit/model issues.`ds4-server --trace`\n\nwrites the rendered prompts, cache decisions, generated text, and tool-parser events for a whole agent session.\n\nThe DwarfStar logo was designed by hand by Salvatore Sanfilippo, made more graphical with AI, and manually reworked by Ben Gnomino, whose human touch made it rock.", "url": "https://wpnews.pro/news/show-hn-deepseek-v4-flash-with-7-7-gib-ram-using-nvme-demand-paging", "canonical_source": "https://github.com/baker27727/ds4-8gb-cpu", "published_at": "2026-08-11 12:18:40+00:00", "updated_at": "2026-08-11 12:43:23.057942+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research", "developer-tools"], "entities": ["DeepSeek V4 Flash", "Mutaz Abubaker", "ChatGPT (GPT-5.6 Sol)", "DwarfStar", "antirez/ds4", "GLM 5.2", "DeepSeek V4 PRO", "llama.cpp"], "alternates": {"html": "https://wpnews.pro/news/show-hn-deepseek-v4-flash-with-7-7-gib-ram-using-nvme-demand-paging", "markdown": "https://wpnews.pro/news/show-hn-deepseek-v4-flash-with-7-7-gib-ram-using-nvme-demand-paging.md", "text": "https://wpnews.pro/news/show-hn-deepseek-v4-flash-with-7-7-gib-ram-using-nvme-demand-paging.txt", "jsonld": "https://wpnews.pro/news/show-hn-deepseek-v4-flash-with-7-7-gib-ram-using-nvme-demand-paging.jsonld"}}