Slotstream: Swift, SSD Expert Streaming, and Breaking the LLM Memory Wall

Independent developer Carlos Galarza released slotstream, a Swift binary that streams Mixture-of-Experts model tensors from NVMe SSD into Apple Silicon Metal memory, enabling local inference of Alibaba's 125-billion-parameter Qwen3.8-Flash-Next model on a 48 GB M5 Pro Mac without requiring dual NVIDIA A100 GPUs or a $6,000 workstation. The tool allocates a 33 GB memory pool with 7,280 expert slots, evicting least-recently-used experts to keep only active parameters in RAM.

The Catalyst & The Problem: Bypassing the VRAM Monopoly When Alibaba released Qwen3.8-Flash-Next—a 125-billion-parameter Mixture-of-Experts MoE preview of the Qwen4 architecture—it highlighted a persistent dilemma in open-source AI: memory wall constraints. At 4-bit quantization, the model weighs 104 gigabytes on disk. For hardware enthusiasts and local inference advocates, running a model of this magnitude historically required an enterprise rig with dual NVIDIA A100 GPUs or a maxed-out Apple Silicon workstation costing upwards of six thousand dollars. Traditional local runtimes like llama.cpp or PyTorch-backed Python servers rely on memory-mapping mmap or pinning the full weight tensor graph in RAM before execution begins. When a model exceeds physical RAM, operating system page thrashing collapses token generation speed from tens of tokens per second to sub-fractional crawling speeds. Python-based runtimes add secondary layers of memory fragmentation, GIL bottlenecks, and bulky IPC serialization overhead. Carlos Galarza carloslfu , an indie developer focused on executable rationality and low-level software efficiency, saw an architectural opening. Mixture-of-Experts models do not activate all parameters simultaneously for every token. In Qwen3.8-Flash-Next, while 125B total parameters exist across 512 total experts, only a small fraction around 6B parameters are activated during any single forward pass token evaluation. Galarza posed a fundamental engineering question: If 90% of an MoE model sits idle during any given matrix multiplication step, why keep all 104 gigabytes pinned in unified memory? The result is slotstream , a lightweight, standalone Swift binary designed specifically to stream MoE expert tensors on demand from fast NVMe SSD storage into tightly budgeted Apple Silicon Metal memory buffers. Under the Hood: SSD Expert Streaming and Metal Slot Allocation At the core of slotstream is a custom expert cache manager and an asynchronous IO engine built on top of Apple’s Swift language and MLX framework bindings. Rather than relying on OS-managed page faults, slotstream controls tensor lifetimes through an explicit, bounded slot array allocator. 1. Dynamic Expert Slot Pool When slotstream initializes, it scans available system memory and allocates a fixed memory pool. On a 48 GB M5 Pro Mac, it claims approximately 33 GB of memory, organizing this space into 7,280 global expert slots ~20.1 GB dedicated purely to expert weight caches, with the remainder reserved for attention layers, base model weights, and KV prefix caches . Each layer in the transformer stack dynamically routes incoming tokens to top-k experts. When a layer requests expert E i, the slot engine checks if E i already resides in the slot cache. If present, execution proceeds zero-copy in Apple Silicon unified memory. If missing, the engine evicts the Least Recently Used LRU expert slot and issues a direct, unbuffered asynchronous read command to the APFS storage system. Here is a conceptual Swift representation of slotstream's lock-free expert ring buffer and eviction pipeline: js final class ExpertSlotManager { private struct Slot { var expertID: Int32 var lastAccessedStep: UInt64 var metalBuffer: MTLBuffer } private var slotPool: Slot private var activeMapping: Int32: Int // Expert ID to Pool Index private let ssdStreamer: AsyncBlockReader init capacity: Int, device: MTLDevice { self.slotPool = 0..<capacity .map { in Slot expertID: -1, lastAccessedStep: 0, metalBuffer: device.makeBuffer length: EXPERT BYTE SIZE, options: .storageModeShared } self.activeMapping = : } func requestExpert id: Int32, step: UInt64 async throws - MTLBuffer { if let slotIndex = activeMapping id { slotPool slotIndex .lastAccessedStep = step return slotPool slotIndex .metalBuffer } // LRU Eviction Logic let lruIndex = slotPool.indices.min by: { slotPool $0 .lastAccessedStep < slotPool $1 .lastAccessedStep } let evictedExpert = slotPool lruIndex .expertID if evictedExpert = -1 { activeMapping.removeValue forKey: evictedExpert } // Direct Async Read from SSD directly into Unified RAM buffer try await ssdStreamer.readExpertWeights id: id, into: slotPool lruIndex .metalBuffer slotPool lruIndex .expertID = id slotPool lruIndex .lastAccessedStep = step activeMapping id = lruIndex return slotPool lruIndex .metalBuffer } } 2. Metal Unified Memory & Zero-Copy Alignment Because Apple Silicon features unified architecture where CPU and GPU share the same physical LPDDR DRAM bus, slotstream avoids CPU-to-GPU PCI Express copying entirely. Reads performed via APFS asynchronous disk IO write straight into unified memory mapped as .storageModeShared Metal buffers. The Metal compute pipeline immediately executes matrix multiplications GEMM on those exact byte addresses without intermediate serialization. 3. Speculative Multi-Token Prediction MTP For machines with memory to spare target allocation above 26 GB , slotstream implements speculative decoding via an auxiliary Multi-Token Prediction --mtp draft head. The draft head predicts the subsequent token in parallel with the target token. Slotstream drafts multiple tokens forward and validates them in a single batched verification pass. Benchmarks demonstrate an 86% draft acceptance rate on the first draft pass, boosting warm generation speeds up to ~12 tokens per second on an M5 Pro without incurring exponential IO overhead. 4. Prefix KV Cache Reuse Across Conversations To overcome the high prefill latency associated with streaming context parameters, slotstream implements prefix KV cache persistence across conversation turns. During turn-based chat sessions, follow-up messages re-use cached key-value states up to 32,768 tokens. Rather than re-computing 8,000 tokens of chat history—which takes up to 60 seconds on a 48 GB machine—subsequent turns execute in a flat 6.0 seconds. Hands-on Quickstart: Deploying Qwen3.8-Flash-Next in Seconds Getting slotstream up and running on macOS requires no Python virtual environments, C++ compiler flags, or Xcode installations. Command Line Tools are sufficient. Installation Execute the self-contained installation script, which places a signed Swift binary into ~/.slotstream/bin : curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh Alternatively, compile directly from source: git clone https://github.com/carloslfu/slotstream && cd slotstream make build Pre-flight Diagnostics Before pulling the 104 GB model checkpoint, run the hardware diagnostic tool to verify system RAM, disk headroom, and projected decode performance: slotstream doctor To simulate performance under lower memory constraints e.g., assessing behavior on a 16 GB MacBook Air , pass the simulation flag: slotstream doctor --sim-ram 16 Executing CLI Prompts & API Server Run a quick single-turn prompt directly from the shell: slotstream run --prompt "Explain the mechanics of zero-copy disk streaming in OS kernels." To integrate slotstream with existing LLM tools like Open WebUI, desktop apps, or custom agent scripts, start the embedded background server listening on Ollama’s default port 11434 : slotstream serve Now communicate using standard curl commands or the standard Ollama CLI: curl http://localhost:11434/api/chat -d '{ "model": "qwen3.8-flash-next:4bit", "messages": {"role": "user", "content": "Write a lock-free queue in C++20."} }' Honest Architectural Critique & Trade-offs Slotstream is an inspiring engineering effort, but its architectural decisions introduce clear compromises that developers must understand before adopting it for production workloads. The Praise Sublime Hardware Allocation: The automatic sizing heuristic slotstream memory plan is masterfully calibrated. Taking the minimum of 33 GB, 70% RAM, and 2 GB under the Metal working set prevents kernel panics and out-of-memory crashes gracefully. Zero Python Bloat: Delivering an entire 125B MoE inference engine in a single, lightweight Swift binary eliminates heavy PyTorch dependencies, Conda environments, and multi-gigabyte wheel installations. Deterministic Integrity: Downloads are verified against SHA-256 hashes hardcoded into the binary assets, backed by signed GitHub release attestations gh attestation verify . The Criticism SSD Write/Read Thermal & Lifespan Stress: Continuous streaming of 104 GB model weights across long inference sessions reads gigabytes of raw data per minute from the system drive. On fanless devices or thin MacBooks, sustained high NVMe reads trigger thermal throttling and accelerate SSD wear over extended usage cycles. Severe Initial Prefill Latency: Cold prefill processing remains an undeniable bottleneck. Processing an 8,000-token prompt requires nearly 60 seconds on a 48 GB Mac and over 3 minutes on a 16 GB machine because attention prefill must process all expert layers sequentially before generating token number one. Feature Set Restrictions: Slotstream intentionally truncates modern API features. Vision inputs, function calling/tools, structured JSON-schema enforcement, and log probabilities are currently omitted, returning an explicit HTTP 400 error. The Alternative Landscape To understand where slotstream fits, we must compare it against existing local LLM runtimes. | Framework | Primary Language | 125B MoE Strategy | Minimum RAM for 104GB Model | Warm Generation 48GB Mac | |---|---|---|---|---| slotstream | Swift + MLX | Dynamic SSD expert slot streaming into Metal ring buffers | 8 GB 16 GB recommended | ~12 tok/s | llama.cpp | C++ / Metal | mmap page swapping / static CPU-GPU split | 64 GB+ thrashing below this limit | ~1-3 tok/s when thrashing | Ollama | Go / C++ | Whole-model VRAM loading via llama.cpp backend | 96 GB+ | N/A OOM on <64GB | MLX-LM | Python / C++ | In-memory tensor graph allocation | 104 GB+ | N/A OOM on <96GB | While llama.cpp remains the swiss-army knife of local LLM quantization, its reliance on generic OS virtual memory paging makes MoE expert offloading inefficient when physical RAM is strictly constrained. MLX-LM offers outstanding Python-native speed for models that fit entirely in unified RAM, but fails instantly when model size exceeds system capacity. Slotstream occupies a specialized niche: sacrificing peak compute throughput during prefill to achieve remarkable execution stability on constrained hardware. The Author & The Open-Source Movement Carlos Galarza carloslfu represents a growing movement of indie systems hackers who reject the assumption that modern AI software must be wrapped in heavy Python layers and microservice bloat. Working under the banner of "Executable Rationality," Galarza’s work emphasizes clean software architectures, zero-dependency native binaries, and mathematical optimization over raw compute brute-forcing. Projects like slotstream highlight a fundamental shift in local AI engineering. As Mixture-of-Experts architectures like Qwen3.8-Flash-Next become the modern standard for frontier performance, software engineers are realizing that memory bandwidth and storage latency—rather than pure FLOPs—are the defining bottlenecks of edge intelligence. By turning Swift into a high-performance system execution language for Apple Silicon, slotstream demonstrates that low-level hardware alignment can unlock frontier AI models for millions of developers who previously thought their machines were too small to join the open-source intelligence wave.