Edge0-145B is a 145B-parameter latent MoE model built for 4-bit native inference. Here's what it takes to run it on one GPU.
What is Edge0-145B? #
Edge0-145B is a 145-billion-parameter language model built around a sparse “latent MoE” (mixture of experts) architecture and trained natively at 4-bit precision rather than quantized down after the fact. The pitch behind it is simple: a model with 145 billion total parameters that can still fit and run on a single 24GB consumer GPU, like an RTX 5090 or RTX 4090, instead of requiring a multi-GPU server rack. It ships as open weights on Hugging Face in two main forms: an NVFP4 build and a Q3_K GGUF-style quantization, both aimed at people who want frontier-scale parameter counts without data-center hardware.
TL;DR #
- Edge0-145B packs 145 billion parameters into a sparse mixture-of-experts design, but because it’s trained natively at 4-bit precision, only a fraction of those parameters are active and resident in VRAM at once.
- The NVFP4 build targets modern Blackwell-class cards like the RTX 5090, taking advantage of native 4-bit floating point support for higher throughput.
- The Q3_K quantization is a more aggressive compression aimed at older or smaller-VRAM cards, including the RTX 4090, trading some quality for a lower memory footprint.
- Both releases are distributed as open weights on Hugging Face, with the NVFP4 version and Q3_K version tracked as separate repositories under the Edge0 organization.
- The project is actively used : the NVFP4 repo shows over 55,000 downloads and 467 likes, while the Q3_K repo shows over 92,000 downloads and 136 likes, suggesting the smaller quant is the more popular on-ramp for local use.
- Apple Silicon, specifically the M2 Ultra , is referenced as a viable platform for running the model given its unified memory pool, which sidesteps the discrete VRAM ceiling that constrains most consumer GPUs.
- A GitHub repository (Edge0-AI/Edge0) accompanies the model weights, pointing to inference tooling and setup instructions rather than requiring users to build a serving stack from scratch.
Enterprises have Remy.
How does a 145B model fit on one GPU? #
The trick is sparsity combined with native low-bit training. A dense 145B-parameter model stored at even 8-bit precision would need well over 140GB of memory just to hold the weights, before accounting for activations or context. Edge0-145B avoids that wall two ways.
First, it’s a mixture-of-experts model. Only a subset of the total parameters activate for any given token, so the effective compute and memory traffic per step is much lower than the headline parameter count implies. This is the same general principle behind other large MoE releases: the “145B” figure describes total capacity, not what’s doing the work on every forward pass.
Second, Edge0-145B is trained natively at 4-bit precision instead of being trained in 16-bit and quantized afterward. Post-hoc quantization often degrades quality because the model never learned to operate at that precision. Training natively at 4-bit means the weight distributions are shaped for low-bit storage from the start, which is the stated rationale for why the NVFP4 build can run with a smaller quality hit than a typical quantized-after-the-fact model of similar size.
Put together, a sparse architecture plus native 4-bit weights is what lets the full checkpoint land within the 24GB VRAM budget of a single consumer card.
What’s the difference between the NVFP4 and Q3_K builds? #
NVFP4 is a 4-bit floating point format that newer NVIDIA GPUs (the Blackwell generation, including the RTX 5090) support natively in hardware. Running a model in the format it was trained in, on hardware built to accelerate that exact format, is the best-case scenario for both speed and output quality. This is the build to reach for if your GPU supports it.
Q3_K is a different approach: it’s a further quantization down to roughly 3 bits per weight, using the K-quant scheme common in the GGUF ecosystem (the format used by llama.cpp and similar local-inference tools). Q3_K trades some precision for a smaller file size and lower VRAM requirement, which matters if you’re on an RTX 4090 or another card without native FP4 acceleration, or if you simply want more headroom for longer context windows alongside the model weights.
In practice, this gives users a choice along a quality-versus-compatibility axis: NVFP4 for newer hardware and best fidelity, Q3_K for broader hardware support and a lighter footprint.
What hardware do you actually need? #
The model card frames three reference platforms:
- RTX 5090 (24GB) : the primary target for the NVFP4 build, pairing native 4-bit hardware support with enough VRAM to hold the model and run inference at usable speed.
- RTX 4090 (24GB) : compatible with both builds, though the Q3_K quantization is the more realistic fit given the 4090 lacks native FP4 acceleration.
- Apple M2 Ultra : relevant because of its large unified memory pool, which lets the system allocate memory flexibly between the model and other workloads rather than being capped by a fixed discrete VRAM pool the way GPUs are.
Enterprises use Remy.
ChatGPT Here’s a draft of your policy…
Gemini Here are 5 ways to track it…
Perplexity According to 4 sources…
-
Self-hosted
-
SOC 2
-
HIPAA
-
Audit log
-
Role-based access The common thread across all three is that none of them are data-center hardware. They’re high-end consumer or prosumer machines that an individual developer or small team can plausibly already own, which is the core claim behind Edge0-145B’s design: a 145B-class model that doesn’t require a cluster to touch.
Is Edge0-145B worth running locally? #
For developers who want to experiment with a very large open-weight model without renting cloud GPU time, Edge0-145B is worth a look specifically because of the hardware story. Most models with parameter counts in this range require multiple high-memory GPUs or quantization schemes aggressive enough to noticeably hurt output quality. Edge0-145B’s native 4-bit training is meant to close that gap, and the two build options (NVFP4 and Q3_K) give you a way to match the model to whatever GPU you have rather than being locked out entirely. The tradeoffs to weigh: a 3-bit quantization like Q3_K will cost some quality relative to the NVFP4 build, and the NVFP4 build’s performance advantage depends on having hardware with native support for that format. If you’re on an older GPU with less than 24GB of VRAM, neither build is likely to fit comfortably, and you’d be looking at further quantization or off some layers to system RAM, which slows things down substantially.
The download numbers on Hugging Face (over 55,000 for NVFP4, over 92,000 for Q3_K) suggest a real and active user base rather than a niche research curiosity, which is a reasonable signal that the setup process works in practice for people running consumer hardware.
How do you get started running it? #
The model weights are distributed as open-weight releases on Hugging Face under the Edge0 organization, with NVFP4 and Q3_K as separate repositories. A companion GitHub repository (Edge0-AI/Edge0) hosts the inference code and setup instructions needed to actually load and serve the model, rather than expecting users to wire up their own serving stack from raw weights.
The general workflow mirrors other local open-weight LLM setups: clone the inference repo, download the weights for whichever quantization matches your hardware, and point the provided runner at your GPU. Users on NVFP4-capable hardware should pull that repository; everyone else, including most RTX 4090 owners and Apple Silicon users, will likely get better results starting with the Q3_K build.
Frequently Asked Questions #
How much VRAM does Edge0-145B need?
The model is designed to fit within a single 24GB consumer GPU, such as the RTX 5090 or RTX 4090, when using either the NVFP4 or Q3_K quantized build. Exact usable context length and headroom for other processes will vary depending on which build and how much of the 24GB is otherwise free.
What does “latent MoE” mean in this context?
It refers to a mixture-of-experts architecture where only a subset of the model’s total parameters are active for any given input. This is what allows a model with 145 billion total parameters to run with a far smaller active compute and memory footprint than a dense model of the same size would require.
Is NVFP4 better than Q3_K?
Nothing gets lost.
312 apps across 41 teams · all in one place
NVFP4 generally preserves more quality because it matches the precision the model was natively trained at and runs on hardware (newer NVIDIA GPUs) built to accelerate that exact format. Q3_K compresses further, to around 3 bits per weight, which lowers VRAM needs and broadens hardware compatibility but introduces more quality loss.
Can I run Edge0-145B on a Mac?
The model card references the Apple M2 Ultra as a supported platform, leaning on its large unified memory pool rather than a fixed discrete VRAM allocation. This makes Apple Silicon with sufficient unified memory a viable alternative to a dedicated GPU setup.
Where do I download Edge0-145B?
Both quantized builds are published as open weights on Hugging Face under the Edge0 organization (separate repositories for NVFP4 and Q3_K), with an accompanying GitHub repository, Edge0-AI/Edge0, providing the inference tooling and setup instructions.