Run 125B+ MoE on consumer GPUs

A project called Strata, from developer Niko1221, claims to run the 125-billion-parameter Qwen3.8-Flash-Next mixture-of-experts model on a single 12GB NVIDIA GPU plus 64GB of system RAM by streaming only active expert weights over PCIe. Strata's own benchmark table reports 21.4 tokens/sec on an RTX 4070 with 64GB DDR5 and 52GB RAM allocation, versus out-of-memory failure for standard Ollama and 3.8 tokens/sec for vLLM with CPU offload. The engine uses predictive sparse routing that anticipates which of the 64 experts fire 1-2 tokens ahead, a tiered VRAM-RAM cache, and zero-copy pinned memory over asynchronous CUDA streams, with the code distributed via the GitHub repository Niko1221/Strata.

RUN 125B+ MoE ON CONSUMER GPUS Powered by Niko1221's dynamic sparse router offloading. Run server-grade frontier models like Qwen3.8-Flash-Next 125B on a single 12GB NVIDIA graphics card and 64GB system RAM—with zero cloud dependencies. git clone https://github.com/Niko1221/Strata.git cd Strata .\START-HERE.bat How Strata Solves the VRAM Wall Traditional inference engines crash when a 125B model exceeds GPU memory. Strata utilizes the sparse nature of Mixture-of-Experts architectures to stream only active weights: Predictive Sparse Routing In MoE models like Qwen 125B, only 2-8 out of 64 experts fire per token. Strata predicts which expert layers will be called 1-2 tokens in advance, queuing them for instant DMA PCIe transfer. Tiered VRAM-RAM Cache Critical attention heads and frequently triggered experts reside permanently inside your 12GB GPU VRAM. The remaining 90GB of dormant weights stream through high-speed 64GB DDR5 memory. Zero-Copy Pinned Memory Weights are transferred over asynchronous CUDA streams via PCIe 4.0/5.0 directly into tensor cores, completely bypassing CPU compute cycles and preventing stutter. System Requirements Matrix NVIDIA RTX 3060 / 4060 / 3070 12 GB VRAM 64 GB DDR4/DDR5 Dual Channel PCIe 3.0/4.0 NVMe SSD 50GB+ Free 12 - 18 Tokens/sec MoE Offloaded Recommended Models: - Qwen3.8-Flash-Next 125B 4-bit Q K - DeepSeek-V2-Lite 16B NVIDIA RTX 4070 Ti / 4080 / 3090 16 GB - 24 GB VRAM 64 GB - 128 GB DDR5 PCIe 4.0 NVMe SSD 7000 MB/s read 25 - 38 Tokens/sec Recommended Models: - Qwen3.8-Flash-Next 125B 5-bit Q M - Mixtral 8x22B Instruct NVIDIA RTX 4090 24 GB or Dual RTX 3090 24 GB - 48 GB VRAM 128 GB DDR5 Quad Channel PCIe 5.0 / Gen4 NVMe DirectStorage 45 - 65 Tokens/sec Recommended Models: - Full-context 125B-236B MoE Models - Continuous Batching Serving Engine Performance Comparison Testing Qwen3.8-Flash-Next 125 Billion Parameters on a standard single RTX 4070 12GB VRAM paired with 64GB DDR5: | Inference Engine | Hardware Target | Tokens / Sec | RAM Allocation | Status & Stability | |---|---|---|---|---| | Strata LLM Niko1221 | RTX 4070 12GB + 64GB RAM | 21.4 tok/s | 52 GB | Native Support Zero OOM Crash | | Ollama Standard | RTX 4070 12GB + 64GB RAM | Fail / OOM | Exceeds limit | Out of Memory / Requires Quantization Drop | | vLLM CPU-Offload | RTX 4070 12GB + 64GB RAM | 3.8 tok/s | 58 GB | Severe PCIe Bus Bottleneck | Frequently Asked Questions Everything you need to know about setting up Strata and configuring MoE quantization.