Edge0-145B: How a 145B-Parameter Model Fits on One RTX 5090

Edge0 released Edge0-145B, a 145-billion-parameter sparse mixture-of-experts model that runs within 24GB of VRAM, the capacity of a single RTX 5090, by using NVFP4 native 4-bit quantization applied during training and a "latent MoE" architecture that activates only a sparse expert subset per token. The model, built on a Qwen3.8-Flash-Next base and published on Hugging Face with GGUF weights for local inference, targets coding and agentic tasks and has drawn 227 likes, more than 14,000 downloads and a trending score of 200. The repo ships a GGUF skeleton, separate expert and dense weight files, a multi-token-prediction (MTP) module and a start script, and sibling badges reference Edge0-120B variants in NVFP4 and q3k quantization.

Edge0-145B: How a 145B-Parameter Model Fits on One RTX 5090 Edge0-145B packs 145 billion parameters into 24GB VRAM using native 4-bit quantization and a sparse MoE design. Here's how it works. What is Edge0-145B? Edge0-145B is a 145-billion-parameter sparse mixture-of-experts MoE model released by Edge0 that runs on a single consumer GPU instead of a multi-GPU server rack. It achieves this through NVFP4, a native 4-bit quantization format baked in during training rather than applied after the fact, combined with a “latent MoE” architecture that only activates a small slice of its parameters per token. The result is a model with the scale of a frontier-class system that fits inside 24GB of VRAM, the amount available on an RTX 5090. TL;DR - Edge0-145B is a 145B-parameter sparse MoE model built on a Qwen3.8-Flash-Next base, released by Edge0 on Hugging Face with GGUF weights for local inference. - The model uses native 4-bit training NVFP4 rather than post-training quantization, which is why Edge0 frames it as “trained natively at 4 bits” instead of compressed down to 4 bits after the fact. - Its latent MoE architecture keeps most of the 145B parameters dormant for any given token, activating only a sparse expert subset, which is what makes the memory footprint manageable. - The release targets coding and agentic tasks , with tags for “agent,” “coding,” and “edge-inference” baked into the model card. - As of its Hugging Face listing, the model has racked up 227 likes and over 14,000 downloads , with a trending score of 200, suggesting fast uptake among local-inference enthusiasts. - The repo ships with a GGUF skeleton, separate expert and dense weight files, a multi-token-prediction MTP module, and a start script , pointing to a purpose-built runtime rather than a drop-in llama.cpp conversion. - Sibling badges in the repo reference Edge0-120B variants in both NVFP4 and q3k quantization , suggesting Edge0 is shipping a family of sizes rather than a single model. Enterprises have Remy. ⌂ Self-hosted ✓ SOC 2 ✚ HIPAA ☰ Audit log How does Edge0-145B fit a 145B model into 24GB of VRAM? The short answer is sparsity plus native low-bit weights. A 145B-parameter dense model stored at 16-bit precision would need roughly 290GB of memory just for weights, far beyond any single consumer card. Edge0 closes that gap two ways. First, NVFP4 quantization stores weights at 4 bits instead of 16, which alone would shrink that 290GB footprint to somewhere around 70-80GB depending on overhead. Crucially, Edge0 describes this as a model “trained natively at 4 bits,” meaning the low-precision representation isn’t bolted on after training the common approach with most quantized community releases but is part of how the model learned its weights in the first place. That generally preserves more quality at a given bit-width than post-hoc quantization schemes. Second, and more important for the 24GB target, Edge0-145B is a sparse MoE model with what the model card calls a “latent MoE” design. In a mixture-of-experts architecture, the full parameter count exists on disk, but only a fraction of those parameters, the “activated” experts, actually run for any given token. The rest sit in memory unused until a different token routes to a different expert. This is the same basic principle behind other large sparse models, but Edge0’s implementation separate experts.bin , dense.bin , and labels.bin files in the repo, alongside a dedicated expert-routing scheme is structured specifically to keep the active working set small enough for a single GPU’s VRAM while the rest can be handled efficiently. The combination of native 4-bit weights and sparse activation is what lets a 145B-parameter model claim a 24GB VRAM footprint, the ceiling on cards like the RTX 5090, rather than requiring the 80GB-plus data-center GPUs usually associated with models of this scale. What’s inside the Edge0-145B release? The Hugging Face repository is unusually granular for a model release, which hints at a custom inference stack rather than a standard transformers checkpoint. Notable files include: - model-skeleton.gguf and ple-fp8.gguf , GGUF-format files for integration with llama.cpp-style runtimes. - experts.bin , dense.bin , and labels.bin , which separate the MoE expert weights, dense always-active layers, and routing labels into distinct binaries. - latent dense q4.bin and latent dense q8.bin , offering two precision options for the dense latent component. - An mtp/ directory containing dense.bin , experts.bin , and draft vocab.bin , indicating a multi-token prediction module, a speculative-decoding-style technique used to speed up generation by predicting several tokens ahead. - native experts.txt , profile.bin , and release-scope.json , suggesting configuration metadata tied to how experts are selected and which parts of the model ship in this particular release. - A start.sh script, implying a packaged way to launch inference rather than relying purely on third-party tooling. The model’s base is listed as Qwen/Qwen3.8-Flash-Next , meaning Edge0 built its MoE and quantization work on top of an existing Qwen architecture rather than training a foundation model from scratch. The license is listed as “other,” with a separate Qwen license file included for the base model component, so anyone deploying it commercially should check both license terms. › Build a shift-handoff app for 900 nurses. already included: ✓ HIPAA✓ access control✓ audit log✓ self-hosted Why does native 4-bit quantization matter more than post-hoc compression? Most “4-bit” open models you’ll find on Hugging Face started life at 16-bit precision and were quantized afterward using methods like GPTQ, AWQ, or GGUF’s own quantization schemes. That process is fast and cheap, but it trades away some accuracy because the model’s weights were never optimized with 4-bit constraints in mind. Training natively at 4 bits, as Edge0 claims for this model, means the weight distributions and gradient updates during training already account for the lower precision. In principle this closes the gap between a quantized model and its full-precision counterpart, because the model isn’t being forced into a representation it wasn’t designed for after the fact. NVFP4 itself is a 4-bit floating-point format distinct from integer-based 4-bit schemes , which generally handles a wider dynamic range of values than fixed-point INT4, reducing the kind of outlier-related errors that plague aggressive quantization. For builders, the practical upshot is that a native 4-bit MoE model has a better shot at matching the task performance of a larger dense model at the same memory budget, compared to taking an existing dense model and quantizing it down after the fact. Is Edge0-145B worth running locally? For anyone already running local LLMs on a single high-VRAM consumer card, Edge0-145B is notable simply because it exists in that size class at all. Most 100B+ parameter models require either multi-GPU setups, cloud inference, or aggressive quantization that noticeably degrades output quality. Edge0 is positioning this release specifically for coding and agentic workloads, based on the “agent” and “coding” tags attached to the model card, and the inclusion of a multi-token prediction module suggests attention to generation speed, which matters for agentic loops that chain many model calls together. The download and engagement numbers on Hugging Face 227 likes, over 14,000 downloads, a trending score of 200 at time of listing indicate real uptake from the local-inference community, not just a research curiosity. Whether it’s “worth it” depends on your hardware: if you already own a 24GB-class card like the RTX 5090, this is one of very few ways to run a model at this parameter scale without renting cloud GPUs. If you’re on an 8GB or 12GB card, the VRAM math won’t work regardless of the quantization tricks. The sibling badges referencing Edge0-120B variants in both NVFP4 and q3k formats suggest Edge0 is building out a lineup rather than shipping a one-off, which is worth watching if the 145B variant proves out in independent benchmarking. Frequently Asked Questions What hardware do you need to run Edge0-145B? Edge0 designed the model to fit within 24GB of VRAM, the amount available on an RTX 5090, by combining native 4-bit NVFP4 quantization with a sparse mixture-of-experts architecture that only activates a fraction of its 145 billion parameters per token. What base model is Edge0-145B built on? The Hugging Face model card lists Qwen/Qwen3.8-Flash-Next as the base model, meaning Edge0’s MoE restructuring and quantization work was applied on top of that existing Qwen architecture. What is NVFP4 quantization? Build an onboarding app for 4,000 new hires. Build ↵ Self-hosted SOC 2 Role-based access Audit log NVFP4 is a 4-bit floating-point format. Edge0-145B was trained natively at this precision rather than quantized down after full-precision training, which generally preserves more accuracy than post-hoc 4-bit compression methods like GPTQ or AWQ. What is a “latent MoE” architecture? It’s Edge0’s term for a mixture-of-experts design where only a sparse subset of the model’s total parameters activates for any given input token, keeping the active memory footprint far below the full 145B parameter count. Does Edge0 offer other model sizes besides 145B? The repository includes badge assets referencing Edge0-120B in both NVFP4 and q3k quantization formats, indicating Edge0 is releasing a family of model sizes rather than a single checkpoint.