{"slug": "show-hn-running-104gb-qwen3-8-flash-next-on-48gb-mac-with-at-12-tok-s", "title": "Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s", "summary": "Slotstream, a new Swift binary, lets users run the 104 GB Qwen3.8-Flash-Next model on a 48 GB Mac at about 12 tokens per second by streaming weights from SSD, with peak memory of 32 GB and a cold start to first token in about 3 seconds. The tool, which supports Ollama and OpenAI chat/generate endpoints, requires about 110 GB free disk space and is available for Apple Silicon Macs running macOS 14 or later, with downloads from Hugging Face taking 30–50 minutes on connections of 400 Mbps or faster.", "body_md": "Run **Qwen3.8-Flash-Next** on a Mac that cannot hold it. The model is 104 GB at\n4-bit; slotstream streams it from SSD and runs it in whatever memory you give\nit, down to an 8.1 GB planned floor. One Swift binary with the commonly used\nOllama and OpenAI chat/generate endpoints.\n\n| on a 48 GB Mac | |\n|---|---|\n| Warm decode | ~12 tok/s |\n| Cold start to first token | ~3 s |\n| Peak memory | 32 GB (auto-sized; you can cap it) |\n| Weights on disk | 104 GB |\n\n**Disk is the gate that bites first.** You need ~110 GB free, so a 512 GB Mac\nis the realistic minimum however much memory it has. The weights are a one-time\n104 GB download: well under an hour on a fast connection, several hours on a\nslow one (table [below](#the-104-gb-download)).\n\n| memory | expect |\n|---|---|\n| 8 GB | below the 8.1 GB floor; `doctor` warns that it will page |\n| 16 GB | ~5 tok/s estimated |\n| 24 GB | ~8 tok/s estimated |\n| 32 GB | ~10 tok/s estimated |\n| 48 GB and up | ~12 tok/s — and auto stops at 33 GB here, so the rest of the machine stays yours |\n\nOnly the 48 GB row is measured on real hardware; the rest come from the same\nmeasured curve, and smaller Macs also have slower SSDs. Run\n`slotstream doctor`\n\nto see what your machine would get, and whether you have\nthe disk for the weights, **before** downloading anything.\n\n```\ncurl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh\n```\n\nInstalls a prebuilt binary to `~/.slotstream/bin`\n\nand puts it on your PATH.\nNeeds Apple Silicon and macOS 14+. Re-run the same line to upgrade; uninstall\nwith `rm -rf ~/.slotstream`\n\n.\n\nReleases are built by CI from the tagged commit with signed provenance, so you can check an asset yourself rather than trusting the download:\n\n```\ngh attestation verify slotstream-arm64.tar.gz --repo carloslfu/slotstream\n```\n\nOr build it yourself — Command Line Tools are enough, no Xcode needed:\n\n```\ngit clone https://github.com/carloslfu/slotstream && cd slotstream\nmake build\n```\n\nThe binary is small; the weights are not. 103.8 GB across 24 files, one time.\n`serve`\n\nand `run`\n\noffer the download on first run, and `slotstream pull`\n\ndoes\nit on its own:\n\n```\nslotstream serve\n```\n\nEither way it prints the size, the destination and your free disk and waits for a yes before transferring anything, and it refuses outright if the disk cannot hold it.\n\n**Hugging Face is the bottleneck, not your link.** Past four connections it\nplateaus: 4, 8, 16 and 32 all landed in the same 36 to 57 MB/s band, and so did\n`hf_xet`\n\n, Hugging Face's own fastest client, while the same link did 134 MB/s\nto an ordinary host. So past roughly 400 Mbps, more bandwidth\nbuys nothing:\n\n| your connection | wait |\n|---|---|\n| 400 Mbps or faster | 30–50 min — Hugging Face's day, not your link |\n| 200 Mbps | ~1 h 10 |\n| 100 Mbps | ~2 h 20 |\n| 50 Mbps | ~4 h 40 |\n| 25 Mbps | ~9 h |\n\nA real install here took 35 min; the top row is wide because Hugging Face's own throughput moved between sessions. The rows below it are arithmetic over 103.8 GB at your full rated speed, so treat them as best cases.\n\nInterrupting is safe: it resumes at the exact byte it stopped on, and all\n24 files are checked against sha256 hashes compiled into the binary, so a\ntruncated, same-size, or corrupted download cannot reach the engine.\n`pull --verify`\n\nre-hashes an existing copy in under 10 s —\n7.7 s here, hashed in parallel.\n\n`serve`\n\nlistens on port 11434 and implements the chat/generate subset used by\nOllama clients and OpenAI SDKs:\n\n```\ncurl localhost:11434/api/chat -d '{\n  \"model\": \"qwen3.8-flash-next:4bit\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"hello\"}]\n}'\nOLLAMA_HOST=http://localhost:11434 ollama run qwen3.8-flash-next:4bit\n```\n\nOpen WebUI, the Ollama CLI, and the OpenAI SDKs are tested for this subset.\nStreaming, CORS, and the usual sampling options (`temperature`\n\n, `top_p`\n\n,\n`top_k`\n\n, `min_p`\n\n, `presence_penalty`\n\n, `seed`\n\n, `num_predict`\n\n, `stop`\n\n) are all\nsupported. Unsupported semantics such as tools, images, JSON-schema output,\nlogprobs, and alternate model names return a clear 400 instead of being\nsilently ignored.\n\nFollow-up turns in a conversation only prefill what is new, so time to first\ntoken stays flat as a chat grows — measured over eight turns, 6.0 s instead of\nclimbing to 25.8 s. One consequence worth knowing: reusing that state is not\nbit-identical to recomputing it, so a reply can occasionally differ where two\ntokens were nearly tied. `--no-prefix-cache`\n\nturns it off if you need exact\nreproducibility.\n\nPrompt plus completion is capped at 32,768 tokens (`--max-context`\n\n). Long prompts are the\nslow axis: prefill runs at roughly 50 tok/s on a 16 GB Mac and 125 on a 48 GB\none, so an 8,000-token prompt waits somewhere between about a minute and about\nthree before its first token. A per-user lock enforces one model process at a time.\n\nWith no flags slotstream sizes itself to your machine and tells you what it chose. This is a 48 GB Mac — it reads 52 GB because everything here counts in decimal GB, while Apple markets the same memory as 48:\n\n```\nslotstream memory plan (auto)\n  device: 52 GB RAM (36.0 GB reclaimable now), 40.2 GB Metal working set\n  target: 33.0 GB total for this process   (override: --memory-gb N | --max-ram-percent P)\n  cache:  ~152 of 512 experts per layer  (7280 global slots = 20.1 GB pool)\n  expect: ~32.0 GB peak, ~12 tok/s warm decode (est. from M5 Pro anchors)\n  prefill: 4096 tokens per pass (~125 tok/s here; costs ~5.3 GB of the target)\n  reuse:  up to 32768 tokens across 4 conversations (~1.2 GB), so a follow-up turn re-prefills only what is new\n```\n\nIt takes the lowest of three limits: **33 GB**, **70% of RAM**, and the Metal\nworking-set limit, and it sizes down further when other apps are actually\nholding memory rather than swapping them out.\n\n33 GB is the interesting one. It is not politeness, it is the knee: the\nsmallest target where the expert cache clears the decode plateau *and* the\nbudget still affords the fast 4,096-token prefill pass. Swept a GB at a time,\nnothing between 34 and 84 GB improves either number. So a 64 GB or 128 GB Mac\nasks for the same 33 GB a 48 GB Mac does — the extra would buy nothing, and\n`doctor`\n\nsays so rather than leaving you to wonder. It\nalso **stays elastic while running**: it re-checks every 15 s and resizes the\ncache between requests, shrinking under pressure and growing back once things\nare calm. Output is byte-identical across resizes.\n\n`--max-ram-percent P`\n\nmoves the 70% share without you having to work out the\nGB. The other two limits still apply, so it can lower the target but not raise\nit past the knee.\n\nThree flags replace auto outright, first one wins, and any of them will go past 33 GB if you want to try it — full expert residency (all 512 per layer, so no routed-expert SSD reads; n-gram rows still stream) needs about 88 GB and has never been measured:\n\n`--memory-gb G`\n\n— total memory for the process. Minimum 8.1.`--experts-per-layer N`\n\n— cache size directly, of the model's 512. Each costs 0.133 GB.`--pool-gb G`\n\n— raw pool size.\n\n`slotstream doctor`\n\nprints the plan any of these would produce, `--sim-ram`\n\n/\n`--sim-available`\n\npreview a different machine entirely, and `--json`\n\nemits the\nplan for scripts with the estimates unrounded.\n\nAlmost all of the model's bytes sit in two places: 68 GB of routed experts\n(512 per layer, 10 active per token) and a 32 GB n-gram table. The dense trunk\nis only 3.8 GB and stays resident. Experts are read with `pread`\n\ninto a fixed\npool of cache slots shared by all 48 layers, so hot layers borrow slots from\ncold ones.\n\n**Cache size changes speed, never output.** Greedy decoding is byte-identical\nbetween a 4 GB cache and a 24 GB one, and that equivalence is a standing test.\n\nWhy not just mmap the file? MLX cannot materialize part of a memory-mapped\ntensor: a top-10 expert gather evaluates all 512 experts of that layer, and a\n16-row n-gram lookup evaluates the whole 250 MB shard, so an mmap path loads\n~100 GB and dies. The stock `mlx_lm.load()`\n\nroute took this 48 GB machine into\n48 GB of swap without producing a token.\n\nWorking, and measured on **one machine** — an M5 Pro with 48 GB. The smaller\ntiers are derived from its curve, not run on real 16 GB hardware.\n\nKnown gaps:\n\n**Long prompts are slow to start.** Everything in the prompt is processed before the first token appears. Prefill is ~10x faster per token than generation (~113 tok/s against ~11), but you pay it for every prompt token up front: a 15-token prompt starts in under 2 s, an 8,000-token one takes about 70. Within a conversation you only pay it once — follow-up turns reuse the previous state. Compute is now the bulk of that time, and closing it means a grouped-GEMM kernel.**macOS 14 and 15** have only had the installer exercised, not the runtime.\n\n[PLAN.md](/carloslfu/slotstream/blob/main/PLAN.md) has the design and the milestone tracker;\n[MEASUREMENTS.md](/carloslfu/slotstream/blob/main/MEASUREMENTS.md) has every number here with its method,\nincluding the experiments that failed.\n\n`Tools/verify.sh`\n\nis the acceptance battery — 81 checks covering weight\nprovenance, goldens against a version-matched Python reference, planner\nbehaviour across simulated machines, byte-equality across cache sizes and live\nresizes, the `--memory-gb`\n\npromise, and a serving-robustness suite of inputs\nthat used to crash the server.\n\n`Tools/e2e_release.sh`\n\nruns 31 more against the *installed* binary from\n`curl | sh`\n\n, which is the thing users actually get.\n\nThe parts that need no weights (planner, sampler vs a numpy reference, governor policy, API robustness) run in CI on every release build.\n\nMIT. `Sources/SlotstreamCore/Vendored/GatedDelta.swift`\n\nis ported from\n[mlx-swift-lm](https://github.com/ml-explore/mlx-swift-lm) (MIT), and\n`Tools/reference/`\n\nvendors the community `qwen4_exp.py`\n\nused as the test\noracle. Weights come from\n[pipenetwork/Qwen3.8-Flash-Next-MLX-4bit](https://huggingface.co/pipenetwork/Qwen3.8-Flash-Next-MLX-4bit)\nand remain under the Qwen community license.", "url": "https://wpnews.pro/news/show-hn-running-104gb-qwen3-8-flash-next-on-48gb-mac-with-at-12-tok-s", "canonical_source": "https://github.com/carloslfu/slotstream", "published_at": "2026-09-01 16:42:46+00:00", "updated_at": "2026-09-01 17:24:19.908560+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "large-language-models"], "entities": ["Slotstream", "Qwen3.8-Flash-Next", "Hugging Face", "Ollama", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/show-hn-running-104gb-qwen3-8-flash-next-on-48gb-mac-with-at-12-tok-s", "markdown": "https://wpnews.pro/news/show-hn-running-104gb-qwen3-8-flash-next-on-48gb-mac-with-at-12-tok-s.md", "text": "https://wpnews.pro/news/show-hn-running-104gb-qwen3-8-flash-next-on-48gb-mac-with-at-12-tok-s.txt", "jsonld": "https://wpnews.pro/news/show-hn-running-104gb-qwen3-8-flash-next-on-48gb-mac-with-at-12-tok-s.jsonld"}}