{"slug": "show-hn-qwen3-6-35b-a3b-on-a-16-gb-m1-pro-with-ssd-streamed-moe", "title": "Show HN: Qwen3.6-35B-A3B on a 16 GB M1 Pro with SSD-streamed MoE", "summary": "A developer forked the DwarfStar inference engine to enable running large Mixture-of-Experts models like Qwen3.6-35B-A3B on 16 GB Apple Silicon Macs by streaming model weights from SSD. The project explores adaptive memory management and Metal GPU residency policies to push specialized local inference beyond conventional RAM limits.", "body_md": "**Specialized local inference for models that do not fit in memory.**\n\nA transparent research and co-development fork of\n[antirez/ds4](https://github.com/antirez/ds4), focused on Metal,\nadaptive SSD streaming, common 16–64 GB Apple Silicon systems, and measured experimentation.\n\n[ Quick start](#quick-start)\n·\n\n[Benchmarks](#measured-results)·\n\n[Models](#model-status)·\n\n[DSBox](https://github.com/andreaborio/dsbox)·\n\n[Upstream diff](https://github.com/antirez/ds4/compare/main...andreaborio:ds4:main)·\n\n[Documentation](#documentation)\n\nImportant\n\nThis is [ andreaborio/ds4](https://github.com/andreaborio/ds4), a fork of\n\n[. It does](https://github.com/antirez/ds4)\n\n`antirez/ds4`\n\n**not** aim to replace upstream. The goal is to co-develop DwarfStar: explore complementary hardware and model paths here, then propose every general, reproducible improvement back to upstream when it clears the correctness and performance bar.\n\nDwarfStar is a small, self-contained inference engine optimized around a narrow\nset of very large models. It includes native model loading, prompt rendering,\ntool calling, RAM/on-disk KV state, an HTTP server, a coding agent, GGUF tooling,\nand correctness and speed tests. It is intentionally **not** a generic GGUF\nrunner; arbitrary GGUF files are not expected to work.\n\nUpstream DwarfStar provides the core engine and leads the high-memory and distributed paths. This fork asks a complementary question:\n\n**How far can the same specialized design be pushed on the 16–64 GB Macs many\ndevelopers already own, when the SSD becomes an active model-memory tier?**\n\nThe current work concentrates on:\n\n- adaptive Metal residency and routed-expert cache policies across memory tiers;\n- SSD streaming that accounts for page cache, wired memory, swap, I/O, and throughput together;\n- safer model-backed experiments near macOS memory limits;\n- measured GLM 5.2 work and Qwen3.6-35B-A3B bring-up;\n- GGUF calibration, incremental quantization, and expert-analysis tooling;\n- keeping useful changes small enough to validate and send upstream.\n\nThis is primarily a learning and systems-research project. The fork lets work continue while upstream changes are under review; it is not a parallel rewrite or a competing inference ecosystem.\n\nThe boundary is explicit and reviewable:\n\n| Change type | Where it belongs |\n|---|---|\n| General, reproducible, backend-safe improvement | Open a PR against\n`antirez/ds4` |\n\nThis is mandatory, not aspirational: every fork change applicable to an upstream-supported path will be opened upstream once its scope, correctness, and performance evidence are ready. Fork development can continue while that review is in progress.\n\nCurrent upstream work includes [#434](https://github.com/antirez/ds4/pull/434)\n(quality-score build fix), [#520](https://github.com/antirez/ds4/pull/520)\n(GLM streamed-prefill correctness), and\n[#528](https://github.com/antirez/ds4/pull/528) (GLM indexed-prefill prepare).\nThe DeepSeek regression found on the GLM line is tracked in\n[#532](https://github.com/antirez/ds4/issues/532). See\n[ FORK_NOTES.md](/andreaborio/ds4/blob/main/FORK_NOTES.md) for the status of each fork change and\n\n[for sync history. The same policy is part of](/andreaborio/ds4/blob/main/MERGE_LOG.md)\n\n`MERGE_LOG.md`\n\n[. The current](/andreaborio/ds4/blob/main/CONTRIBUTING.md)\n\n`CONTRIBUTING.md`\n\n`main`\n\ndelta is always\ninspectable in GitHub's\n[upstream/fork comparison](https://github.com/antirez/ds4/compare/main...andreaborio:ds4:main).\n\nRequirements: Apple Silicon, Xcode Command Line Tools, and enough SSD space for the selected model. A 64 GB Mac is the practical reference tier for DeepSeek Flash streaming; the 16 GB path is an experimental low-memory tier, not a speed guarantee.\n\n```\nxcode-select --install\ngit clone https://github.com/andreaborio/ds4.git\ncd ds4\n\n./download_model.sh q2-imatrix\nmake\n./ds4 --build-info\n./ds4 -m ./ds4flash.gguf --nothink\n```\n\nOn macOS, AUTO residency keeps the model resident when it safely fits. Otherwise it selects SSD streaming and derives an expert-cache budget from the model geometry and live host memory. Force the SSD path only when you need a controlled run:\n\n```\n./ds4 -m ./ds4flash.gguf --ssd-streaming --ctx 32768 --nothink\n```\n\nStart the local API with:\n\n```\n./ds4-server -m ./ds4flash.gguf --ctx 32768\nphp\nflowchart LR\n    GGUF[\"Model GGUF on SSD\"] --> AUTO[\"AUTO memory planner\"]\n    AUTO -->|\"safe fit\"| RES[\"Resident model\"]\n    AUTO -->|\"model exceeds budget\"| STREAM[\"SSD-streamed model\"]\n    STREAM --> FIXED[\"Mapped fixed / non-routed state\"]\n    STREAM --> CACHE[\"Adaptive routed-expert cache\"]\n    GGUF -->|\"cache miss\"| CACHE\n    FIXED --> METAL[\"Metal graph\"]\n    CACHE --> METAL\n    METAL --> TOKEN[\"Next token\"]\n```\n\nThe fixed model state, KV cache, graph scratch, and macOS file-backed cache all need headroom. The routed-expert cache is the variable tier; making it larger can help only until it starts displacing the pages and allocations the rest of the runtime needs.\n\n| Model | Location | Status | Current focus |\n|---|---|---|---|\n| DeepSeek V4 Flash | `main` |\nPrimary supported path | Metal, adaptive SSD streaming, 16–64 GB measurements |\n| DeepSeek V4 PRO | `main` |\nSupported upstream path | High-memory and distributed inference |\n| GLM 5.2 | `codex/glm52-upstream-clean-bench` |\nExperimental branch | Correct streamed prefill and Metal performance on 64 GB |\nQwen3.6-35B-A3B (`qwen35moe` ) |\n`main` |\nSupported opt-in Metal path, model-backed measured | Metal AUTO mapping, live-pressure fallback, strict SSD cache, resident prefill, and parallel resident decode |\n\nDS4 includes the experimental, self-describing `ds4.expert_major.v2`\n\nlayout for\nDeepSeek V4. It stores each layer as adjacent gate/up/down expert records without\nrequantizing and without keeping a second routed-weight copy. Canonical GGUFs\nremain byte-for-byte compatible with every existing backend; native v2 files\ncurrently require a complete local Apple Metal model and fail early elsewhere.\n\nConversion, full byte-level verification, compatibility limits, and the\nmodel-backed promotion gate are in\n[ docs/deepseek-expert-major-v2.md](/andreaborio/ds4/blob/main/docs/deepseek-expert-major-v2.md). Until a\ndated canonical/native benchmark gate is complete, the canonical DeepSeek GGUF\nremains the release reference. The first M5 Pro SSD tranche is recorded in\n\n[. The distinctly named experimental artifact is](/andreaborio/ds4/blob/main/docs/benchmarks/2026-07-17-deepseek-native-expert-major.md)\n\n`docs/benchmarks/2026-07-17-deepseek-native-expert-major.md`\n\n[, with full conversion provenance and compatibility limits in its model card.](https://huggingface.co/andreaborio/DeepSeek-V4-Flash-DS4-ExpertMajor-v2-GGUF)\n\n`DeepSeek-V4-Flash-DS4-ExpertMajor-v2-GGUF`\n\nThe main branch is qualified and measured with one normalized text-only\nartifact. The recommended release download is the single-layout\n`Qwen3.6-35B-A3B-DS4-ExpertMajor-v1-Q4_K_S.gguf`\n\nfrom\n[ andreaborio/Qwen3.6-35B-A3B-DS4-ExpertMajor-v1-GGUF](https://huggingface.co/andreaborio/Qwen3.6-35B-A3B-DS4-ExpertMajor-v1-GGUF).\nIt stores routed weights once in DS4's expert-major order and activates\nautomatically; no sidecar variables are needed. The canonical\n\n`Qwen3.6-35B-A3B-ds4-Q4_K_S.gguf`\n\nremains supported during migration.\nThe release artifact is 20,808,970,240 bytes (only 406,816 bytes larger than\nthe canonical input) with SHA-256\n`fb2b344d49f0c3dfd854cfc11d92ffc873cc93a1d30bf4664e5aea6f1bfef839`\n\n.This is not generic Qwen or arbitrary community-GGUF support. The literal environment guard is the experimental opt-in; Metal, power 100, and AUTO residency are the Apple defaults, but are shown below for reproducibility:\n\n```\nDS4_QWEN_EXPERIMENTAL_METAL=1 ./ds4 \\\n  -m /absolute/path/to/Qwen3.6-35B-A3B-DS4-ExpertMajor-v1-Q4_K_S.gguf \\\n  --metal --power 100 --ctx 8192 --nothink\n```\n\n`ds4.expert_major.v1`\n\nis an explicit DS4 GGUF extension. Other loaders must\nreject this artifact unless they implement the layout; use the canonical file\nfor llama.cpp, MLX, or other runtimes. Format details, conversion commands,\ncompatibility boundaries, and measured parity are in\n[ docs/qwen-expert-major-store.md](/andreaborio/ds4/blob/main/docs/qwen-expert-major-store.md).\n\nQwen AUTO selects the full-model mapped Metal mode only when both the fixed Metal\nworking-set budget and a point-in-time host-memory pressure check pass. Under\npressure it falls back to SSD and lazily grows the routed-expert cache to the\nlargest complete routing tier admitted by the current conservative snapshot.\nAbove 16 GiB the planner independently reserves the 2.50 GiB static page set,\ncontext/runtime memory, and system headroom. On a 16 GiB Mac, AUTO keeps the\ncomplete static charge but lets those unpinned, pageable GGUF pages share system\nheadroom. It selects the largest complete 320-expert cache cycle admitted by\nthe remaining live and platform budgets rather than imposing a fixed low-RAM\nfloor. Bounded file-backed inactive pages receive full credit only while macOS\nreports normal pressure; unknown or elevated pressure retains half-credit and\nfails closed near the boundary. `--resident`\n\nfails unless both admission checks\npass; because pressure can change after the\nsnapshot, this is a conservative admission policy rather than a future-memory\nguarantee. `--ssd-streaming`\n\nremains the reproducible forced-streaming override.\nIn SSD mode Qwen grows its Metal expert cache in 321-expert slabs (about\n0.529 GiB) instead of taking the generic 4 GiB first slab.\n\nHere `resident`\n\nmeans that DS4 maps the complete tensor payload, disables its\nexplicit SSD expert cache, and executes full-tensor Metal kernels. Metal's\nresidency request is a budgeting hint: it neither pre-faults every GGUF page nor\nproves that every page remains physically resident as later pressure changes.\nThat stronger physical-residency claim requires separate runtime measurement.\nAll neural math in the supported Qwen path is on Metal. The CPU still performs\ntokenization, sampling, route readback, cache bookkeeping, and streamed GGUF\nI/O; a CPU+GPU split of layers or experts is not implemented in this path.\n\nThe hard SSD cache floor is 321 complete routed experts (about 0.53 GiB); 640\n(about 1.06 GiB) is a useful controlled small-cache tier. Startup and the\nper-layer path fail closed if the effective locked cache falls below the floor.\nThe runtime has completed model-backed resident and SSD generation on an M5 Pro\nwith 64 GiB, plus bounded SSD generation on a physical M1 Pro with 16 GiB. On\nproduction main `bd62a0b`\n\n, AUTO started with 321 cached experts for prefill and\ngrew toward 2,241 for decode. One cold request completed at 10.56/8.24\nprefill/generation t/s; four subsequent distinct short prompts had medians of\n15.04/9.77 t/s, with normal memory pressure and no new swapouts. This recheck\nused the canonical migration GGUF; the native ExpertMajor v1 artifact was not\ncopied to the 16 GiB host because only 3.6 GB of disk space was free. The older\n4.06/7.03 result remains a conservative 321-expert compatibility floor, not the\ncurrent production speed. See\n[ tests/qwen/README.md](/andreaborio/ds4/blob/main/tests/qwen/README.md) for the exact artifact contract,\nreproducible evidence, and current limitations.\n\nMetal on Apple Silicon is the current proving ground for fork-specific optimization. The inherited CUDA/DGX Spark and ROCm/Strix Halo DeepSeek paths remain supported targets, but a Metal result is not advertised as a Blackwell or Strix Halo result until it is re-measured on that backend.\n\nBest retained local results so far. These rows are not cross-model rankings: each model uses a different artifact, context, and runtime path.\n\n| Model | Best measured setup | Prefill | Generation / decode | Status |\n|---|---|---|---|---|\n| Qwen3.6-35B-A3B Q4_K_S, 20.81 GB | M5 Pro 64 GB, Metal resident | 258.08 t/s |\n57.81 t/s | Controlled DS4 prefill A/B, +23.3% over the previous dispatch; greedy output identical |\n| Qwen3.6-35B-A3B Q4_K_S, 20.81 GB | M5 Pro 64 GB, page-touched resident CLI | 218.30 t/s | 63.94 t/s |\nBest retained real CLI generation number; same rendered prompt and visible continuation as the llama.cpp reference |\n| Qwen3.6-35B-A3B Q4_K_S, 20.81 GB | M1 Pro 16 GB, Metal AUTO to SSD, canonical migration GGUF | 15.04 t/s |\n9.77 t/s |\nWarm median over four distinct short prompts after one cold run; normal pressure, no new swapouts |\n| DeepSeek V4 Flash IQ2XXS, 86.72 GB | M5 Pro 64 GB, Metal SSD streaming | 20.75 t/s | 12.58 t/s | Direct upstream/fork A/B showed parity, not a fork speedup |\n| GLM 5.2 ds4-native GGUF, 244.14 GiB | M5 Pro 64 GB, Metal SSD streaming | 9.15 t/s |\n0.91 t/s | Indexed-prefill prepare A/B; big prefill win, no decode win |\n\nDeepSeek hardware reference bests from the standard `speed-bench`\n\nsweep:\n\n| Host | Model | Prefill | Generation |\n|---|---|---|---|\n| MacBook Pro M5 Max, 128 GB | Flash q2, 11,707-token context | 463.44 t/s | 25.90 t/s |\n| Mac Studio M3 Ultra, 512 GB | Flash q2, 11,709-token context | 468.03 t/s | 27.39 t/s |\n| Mac Studio M3 Ultra, 512 GB | PRO q2, 32,768-token context | 138.82 t/s | 9.56 t/s |\n| DGX Spark GB10, 128 GB | Flash q2, 7,047-token context | 343.81 t/s | 13.75 t/s |\n\nFull commands, samples, and caveats are in\n[ docs/benchmarks/2026-07-15-qwen-ds4-vs-llamacpp.md](/andreaborio/ds4/blob/main/docs/benchmarks/2026-07-15-qwen-ds4-vs-llamacpp.md),\n\n[,](/andreaborio/ds4/blob/main/docs/benchmarks/2026-07-14-m5-pro.md)\n\n`docs/benchmarks/2026-07-14-m5-pro.md`\n\n[, and](/andreaborio/ds4/blob/main/SSD_STREAMING_VERIFICATION.md)\n\n`SSD_STREAMING_VERIFICATION.md`\n\n[.](/andreaborio/ds4/blob/main/docs/ENGINE_REFERENCE.md)\n\n`docs/ENGINE_REFERENCE.md`\n\nMore expert-cache RAM is not automatically faster. On memory-constrained Macs, an oversized cache can evict the file-backed pages SSD streaming needs and make decode slower even when Activity Monitor appears to show free memory. AUTO therefore treats the routed-expert cache as variable and preserves headroom for fixed weights, KV, scratch, Metal allocations, and the macOS page cache.\n\nDuring development, a model-backed test bypassed SSD streaming and attempted to make an 80.76 GiB GGUF resident with a 100,000-token context on a 64 GiB Mac. Global wired memory reached roughly 61.36 GiB before a watchdog kernel panic. Crashing the host is not an acceptable test outcome.\n\nCurrent `main`\n\nincludes hardware-aware AUTO residency, fail-closed cache\nadmission, bounded benchmark guards, and GPU cleanup before model mappings are\nreleased (`1523b26`\n\n). A stricter guard that rejects resident mappings larger\nthan 90% of physical RAM is tested and published on\n`fix/refuse-oversized-resident-maps`\n\nat `06fd005`\n\n, but is **not yet on main**.\nUntil it is merged, it must not be described as a mainline guarantee.\n\n[DSBox](https://github.com/andreaborio/dsbox) is the companion desktop\ninterface, inspired by Unsloth Studio: discover compatible models, manage ds4,\nchat locally, connect coding agents, and observe memory, swap, disk, and token\nthroughput without hand-assembling every command. DSBox is a separate project\nand still a work in progress.\n\n[\n](https://github.com/andreaborio/dsbox)\n\nDSBox is an optional companion UI, maintained in a separate repository.\n\n: complete model, runtime, server, agent, KV-cache, distributed, backend, and debugging guide.`docs/ENGINE_REFERENCE.md`\n\n: experimental Qwen artifact contract, oracle procedure, Metal + SSD commands, measurements, and limits.`tests/qwen/README.md`\n\n: DS4-native GGUF layout, transactional converter, compatibility, and parity evidence.`docs/qwen-expert-major-store.md`\n\n: generic expert-major manifest and separate DeepSeek/GLM qualification plan.`docs/expert-major-v2-roadmap.md`\n\n: upstream-first contribution policy and correctness/performance gates.`CONTRIBUTING.md`\n\n: fork delta and upstreamability ledger.`FORK_NOTES.md`\n\n: upstream synchronization history.`MERGE_LOG.md`\n\n: Metal build identity, AUTO residency, and benchmark promotion gates.`GOLD_METAL_SSD.md`\n\n: independent SSD-streaming verification campaign.`SSD_STREAMING_VERIFICATION.md`\n\n: live, privacy-preserving imatrix collection.`ONEDGE_IMATRIX.md`\n\n: mixed-precision expert streaming design and validation.`STREAMING_MIXED_PRECISION.md`\n\n: expert profiling and prune-mask research.`EXPERT_PRUNE.md`\n\n: GGUF, imatrix, quantization, and quality tooling.`gguf-tools/README.md`\n\n**Detailed fork additions and research notes**\n\nThe sections below preserve the longer design notes for the fork's research\nfeatures. They are not an exhaustive commit count: adaptive residency, cache\nhardening, benchmark guardrails, telemetry, and safe Metal teardown have also\nevolved since the original five-feature summary was written. The authoritative\nper-change ledger is [ FORK_NOTES.md](/andreaborio/ds4/blob/main/FORK_NOTES.md); upstream syncs are recorded\nin\n\n[.](/andreaborio/ds4/blob/main/MERGE_LOG.md)\n\n`MERGE_LOG.md`\n\nThe fork also carries a GLM 5.2 line on\n[ codex/glm52-upstream-clean-bench](https://github.com/andreaborio/ds4/tree/codex/glm52-upstream-clean-bench):\nupstream's\n\n`glm5.2`\n\nbranch (`bd89932`\n\n) plus eleven commits — the streaming prefill\ncorrectness fixes proposed as\n[antirez/ds4#520](https://github.com/antirez/ds4/pull/520)(real-size prompts were failing under\n\n`--ssd-streaming`\n\n; independently validated by a third party on an M4 Max\n128 GB), the indexed-prefill layer-prepare overlap proposed as\n[antirez/ds4#528](https://github.com/antirez/ds4/pull/528)(measured prefill ×1.6-2.0 across a 2048-8192 sweep in the PR, ×2.4-2.5 re-measured on short prompts, decode unchanged, greedy output byte-identical), the ds4-native GLM 5.2 GGUF layout support the line runs on, a copy of the RAM guard (upstreamed separately, see\n\n[), and a set of default-off streaming experiments (router-ahead prefetch, expert prune/profile hooks, virtual resident decode layers). The short-prompt speedup, the regression below and the MTP gate were re-verified independently with paired A/B runs (](/andreaborio/ds4/blob/main/FORK_NOTES.md)\n\n`FORK_NOTES.md`\n\n[); the sweep figures are from](/andreaborio/ds4/blob/main/SSD_STREAMING_VERIFICATION.md)\n\n`SSD_STREAMING_VERIFICATION.md`\n\n[#528](https://github.com/antirez/ds4/pull/528)'s benchmark.\n\nTwo caveats, both measured:\n\n**Upstream's whole**(DeepSeek-V4-Flash IQ2XXS: 7-8 → ~2-3 tok/s on an M5 Pro 64 GB under`glm5.2`\n\nline decodes DeepSeek Flash ~2.8× slower than`main`\n\n`--ssd-streaming`\n\n, first token ~5-7 s; bisected to the first commit of the line, verified twice on separate days). Keep DeepSeek work on`main`\n\n; reported upstream as[antirez/ds4#532](https://github.com/antirez/ds4/issues/532).**Speculative decode (MTP) on streamed GLM is a measured NO-GO**: the`blk.78`\n\nnextn acceptance probe (branch, a reusable measurement tool; GLM 5.2 ds4-native build) reads ~55% acceptance against the ~75% needed to pay for the extra I/O.`feat/glm-mtp-probe`\n\nThe older bring-up branch\n[ wip/glm52-metal64-strict-probe](https://github.com/andreaborio/ds4/tree/wip/glm52-metal64-strict-probe)\npredates this line and is kept as history.\n\nUpstream collects the routed-MoE importance matrix (imatrix) **offline** from a fixed corpus\n(`ds4 --imatrix-dataset … --imatrix-out …`\n\n). This fork lets ** ds4-server collect it from the\nlive prompt stream on the device**, so a quantized model can be\n\n**re-calibrated to its actual workload**, without ever storing a single user prompt. The only artifact is the imatrix: aggregate per-(layer, expert) activation statistics (squared activations + hit counts), a structure that cannot hold prompt text.\n\n```\nds4-server -m model.gguf --imatrix-out edge.dat                  # collect from live traffic\nds4-server -m model.gguf --imatrix-out edge.dat --imatrix-every 128 --imatrix-min-requests 32\n```\n\nDefault **off** (zero behavioral change); opt-in via `--imatrix-out`\n\n, with periodic snapshots\n(`--imatrix-every`\n\n) and a minimum-requests guard (`--imatrix-min-requests`\n\n). Full design,\nwiring, limits and privacy verification in [ ONEDGE_IMATRIX.md](/andreaborio/ds4/blob/main/ONEDGE_IMATRIX.md).\n\nRe-forging a *variant* (say, adding a per-layer Q4 \"boost\" on top of an IQ2 build that used the\nsame imatrix) normally regenerates **every** routed-expert tensor from the FP weights, even the\nones that don't change. But quantization is deterministic in (FP weights, target type, imatrix\nslice), so an unchanged tensor is **byte-identical** to the one already sitting in a prior\nbuild. Recomputing it is pure waste.\n\n`--reuse PRIOR.gguf`\n\ncopies a planned output tensor straight from PRIOR when its **name, target\ntype and shape** match, and quantizes only the tensors that actually changed (the boosted\nlayers, at their new type).\n\n```\n# 1. build the 2-bit base once\ngguf-tools/deepseek4-quantize --hf FP --template base.gguf --imatrix coder.dat \\\n  --out coder-iq2.gguf\n\n# 2. every boost variant reuses the base's unchanged layers, re-quantizing only the boosted ones\ngguf-tools/deepseek4-quantize --hf FP --template base.gguf --imatrix coder.dat \\\n  --reuse coder-iq2.gguf \\\n  --tensor-type blk.30.ffn_gate_exps.weight=q4_k …  --out coder-q4boost.gguf\n```\n\n**Measured** (DeepSeek-V4-Flash, a 6-of-43-layer Q4 boost over an IQ2 base): a full build is\n~80 minutes; the same variant via `--reuse`\n\ntook **5.5 minutes** (1,310 of 1,328 tensors copied,\n18 regenerated), about a **14× speedup**. The output was verified **byte-for-byte identical** to\na from-scratch build across all 1,328 tensors. The fast build is not an approximation, it is the\nsame file.\n\n**Correctness.** Every build stamps a `quantize.reuse_key`\n\nGGUF KV: an fnv1a64 over the\nsafetensors index, each weight shard's size and mtime, the imatrix content, and a template\nstructural salt. `--reuse`\n\ncopies a tensor only when PRIOR's key matches this build **and** the\nper-tensor type and shape match, so a boosted tensor (different target type) is regenerated, and\na stale / foreign / keyless prior (changed weights, imatrix, or recipe) safely falls back to a\nfull quantize. Copied bytes are size-checked against the plan (a hard error on any mismatch),\nand `--reuse`\n\nrefuses to alias `--out`\n\n. This is **not** present in llama.cpp, which always\nrequantizes from the source weights; the closest prior art is splicing GGUF tensors by hand.\n\nChanging the *imatrix* only changes the tensors the\nimatrix actually steers (the routed expert families: the importance vectors re-allocate bits\ninside those tensors). Everything else — attention, shared experts, norms, embeddings, output —\nis byte-identical across builds that share the same FP weights and template. So every build now\nalso stamps `quantize.reuse_key_weights`\n\n: the same fnv1a64 **without** the imatrix folded in.\nWhen PRIOR matches the full key, behavior is unchanged; when it matches only the weights key\n(same weights, different imatrix — the re-calibration case), `--reuse`\n\ncopies the\nimatrix-independent tensors and regenerates only the steered ones:\n\n```\nreuse: PRIOR.gguf shares the weights key (…) but not the imatrix — copying\n       imatrix-independent tensors, regenerating the steered ones\n```\n\nThe dependence test is conservative and mirrors the generators' own imatrix lookups (routed\n`*_exps.*`\n\nfamilies always count as steered; regular tensors are probed with the exact same\nname resolution `generate_regular()`\n\nuses), so over-approximation can only cost an unneeded\nregeneration, never a stale byte. Priors built before this change carry only the old key and\nkeep the old all-or-nothing behavior.\n\nMeasured (DeepSeek-V4-Flash, 1,328 tensors, M5 Pro): a full re-calibration — same recipe,\n`coder.dat`\n\n→ `general.dat`\n\n— copied **1,199 of 1,328 tensors** and regenerated the 129\nrouted-expert tensors with the new imatrix, in **~45 minutes vs ~80 for the full quantize**.\nByte-level verification: 40/40 sampled imatrix-independent tensors identical to the prior,\n16/16 sampled expert tensors changed, tensor tables identical. The change went through an\nadversarial 3-lens review that rejected the first cut (two stale-byte paths, one strict-mode\nabort — all reachable, all fixed before this exercise: the no-imatrix gate, the coverage\nfingerprint, the I32 probe exclusion).\n\nUpstream `--ssd-streaming`\n\nassumes routed-expert tensors are quantized uniformly across\nlayers. A GGUF with a few layers boosted to Q4_K over an IQ2 base (the forgequant boost\nrecipe) failed **every** request under streaming (`model range … is not covered by mapped model views`\n\n) while serving fine with full residency. Two compounding uniformity\nassumptions are fixed: the streaming prefill span set now also maps the exps tensors of\noff-class (\"boosted\") layers, so they are read through mmap'd no-copy views; and the\nsingle-size-class expert cache pre-seeds its slab size at startup and **rejects** off-size\nlayers (which use the mapped path) instead of silently adopting their size and corrupting\nthe slot accounting.\n\nUniform models are verified **byte-identical** under the change (3/3 builds), full-residency\npaths are untouched, and mixed models were validated with the canary benchmark plus entire\neval suites. Full diagnosis, design and behavior guarantees in\n[ STREAMING_MIXED_PRECISION.md](/andreaborio/ds4/blob/main/STREAMING_MIXED_PRECISION.md); reported upstream with\ndiagnosis and workaround in\n\n[antirez/ds4#388](https://github.com/antirez/ds4/issues/388).\n\n**Update (upstream converged):** antirez has since implemented equivalent mixed-precision\nstreaming upstream. After the latest sync this fork **takes upstream's implementation** of\n`weights_streaming_layer_experts_uniform`\n\n(the only merge conflict; the two designs converged) —\nsee [ MERGE_LOG.md](/andreaborio/ds4/blob/main/MERGE_LOG.md). This addition is effectively now upstream.\n\nTwo small, opt-in hooks for studying *which* experts a domain actually needs, used by the\nforgequant layer/expert A/B work:\n\n— the expert profiler (`DS4_EXPERT_PROFILE_FULL`\n\n`ds4_expert_profile_write_layer`\n\n) emits the*full*per-expert ranking instead of the top-16, so a static prune/keep set can be chosen per layer from real routing statistics.— point it at a`DS4_EXPERT_PRUNE_MASK`\n\n`43 × N_EXPERT`\n\ngrid of`'0'/'1'`\n\n(`'1'`\n\n= prune). The mask is applied to the CPU router's`probs`\n\n**before top-k**(masked experts get a large-negative sentinel so they never win), letting each token route to its next-best surviving expert. This measures \"how much of the domain lives in a few experts\" without re-quantizing anything.\n\n```\n# the mask lives in the CPU router, so enable it (streaming-IQ2 path), then prune:\nDS4_METAL_ENABLE_STREAMING_IQ2_CPU_ROUTER=1 DS4_EXPERT_PRUNE_MASK=mask.txt \\\n  ds4 -m coder-iq2.gguf -p \"…\" --ssd-streaming\n# -> \"ds4: expert prune mask ACTIVE (N experts pruned) from mask.txt\"\n```\n\nBoth default **off** (zero behavioral change). The mask affects only routed (non-hash) layers,\nand only when the CPU router is active (streaming-IQ2 or PRO-Q4 paths). Details in\n[ EXPERT_PRUNE.md](/andreaborio/ds4/blob/main/EXPERT_PRUNE.md).\n\nThe long-form guide now lives in\n[ docs/ENGINE_REFERENCE.md](/andreaborio/ds4/blob/main/docs/ENGINE_REFERENCE.md). It covers model\ndownloads, full-resident and SSD-streamed operation, distributed inference,\npower controls, the native agent, benchmarking, capability evaluation, CLI,\nserver/tool calling, disk KV cache, backends, steering, test vectors, and\ndebugging.\n\nKeeping the manual separate makes this README a reviewable landing page while preserving the full operational reference.\n\nDwarfStar is beta software and `ds4-agent`\n\nremains alpha. The core engine and\nupstream direction come from [ antirez/ds4](https://github.com/antirez/ds4).\nThe project also exists thanks to the kernels, formats, and engineering work\nof\n\n[and GGML.](https://github.com/ggml-org/llama.cpp)\n\n`llama.cpp`\n\nReleased under the [MIT license](/andreaborio/ds4/blob/main/LICENSE). Contributions follow the\n[upstream-first policy](/andreaborio/ds4/blob/main/CONTRIBUTING.md).", "url": "https://wpnews.pro/news/show-hn-qwen3-6-35b-a3b-on-a-16-gb-m1-pro-with-ssd-streamed-moe", "canonical_source": "https://github.com/andreaborio/ds4", "published_at": "2026-07-17 23:20:14+00:00", "updated_at": "2026-07-17 23:51:34.035768+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["DwarfStar", "Qwen3.6-35B-A3B", "Apple Silicon", "Metal", "GGUF", "antirez/ds4", "andreaborio/ds4", "DeepSeek"], "alternates": {"html": "https://wpnews.pro/news/show-hn-qwen3-6-35b-a3b-on-a-16-gb-m1-pro-with-ssd-streamed-moe", "markdown": "https://wpnews.pro/news/show-hn-qwen3-6-35b-a3b-on-a-16-gb-m1-pro-with-ssd-streamed-moe.md", "text": "https://wpnews.pro/news/show-hn-qwen3-6-35b-a3b-on-a-16-gb-m1-pro-with-ssd-streamed-moe.txt", "jsonld": "https://wpnews.pro/news/show-hn-qwen3-6-35b-a3b-on-a-16-gb-m1-pro-with-ssd-streamed-moe.jsonld"}}