What it actually takes to self-host Moonshot AI's 2.8T-parameter Kimi K3.1 Pro, from MXFP4 weights to multi-GPU memory math.
What is Kimi K3.1 Pro? #
Kimi K3.1 Pro is Moonshot AI’s open-weight, natively multimodal agentic model, built as a Mixture-of-Experts (MoE) system with 2.8 trillion total parameters and 104 billion activated per token. It’s post-trained from Kimi K3 with large-scale reinforcement learning across hundreds of downstream domains, and it supports a 1,048,576-token context window. The headline detail for anyone thinking about self-hosting it: the model ships with quantization-aware training baked in, using MXFP4 weights and MXFP8 activations rather than full precision.
TL;DR #
- Kimi K3.1 Pro is a 2.8T-parameter MoE model with only 104B parameters activated per forward pass, which is what makes local inference even conceivable.
- The model is natively quantized during training , not after, using MXFP4 for weights and MXFP8 for activations, so you don’t need to find a post-hoc quant to make it runnable.
- Even at 4-bit precision, the raw weight footprint is enormous , landing in the range of roughly 1.4 terabytes before accounting for KV cache, overhead, or context.
- Community quantizations are already pushing lower , including a 2-bit EXL3 build that trims the footprint further at the cost of some accuracy.
- A 1M-token context window adds real memory pressure on top of the weights , especially with the model’s KDA and Gated DSA attention layers.
- This is a multi-GPU, server-class workload , not something that fits on a single consumer card or even a single high-end workstation GPU.
- Most people will interact with K3.1 Pro through the Kimi API or chat interface rather than hosting it, given the hardware bar.
demandsof enterprises.
How big is Kimi K3.1 Pro, really? #
The model card lists 2.8 trillion total parameters spread across 896 experts, with 16 experts selected per token plus 2 shared experts, giving 104 billion activated parameters per forward pass. That activation ratio (roughly 3.7% of total parameters firing per token) is standard MoE design: it keeps compute cost per token manageable while still storing a very large pool of specialized weights.
The architecture has 93 layers total, with just one dense layer and the rest split between two attention mechanisms: 69 layers using Kernelized Dynamic Attention (KDA) and 24 using Gated DeepSeek Sparse Attention (Gated DSA). Moonshot describes the DSA swap as a move away from full-attention Gated MLA layers, aimed specifically at making long-context inference more efficient. That matters because the model supports a context window over one million tokens, and naive full attention at that length would be prohibitively expensive regardless of how the weights are stored.
There’s also a 401-million-parameter vision encoder (MoonViT-V2), since K3.1 Pro handles both text and image inputs. That’s a rounding error next to the 2.8T language backbone, but it does mean a local deployment needs to support multimodal input pipelines, not just text tokenization.
What does MXFP4 quantization actually mean for hosting it? #
The model card specifies “MXFP4 weights / MXFP8 activations (quantization-aware training).” This is different from the more familiar pattern where a lab releases a full-precision (FP16 or BF16) checkpoint and the community quantizes it down afterward with tools like GGUF or AWQ. Moonshot trained K3.1 Pro with quantization in the loop, so the MXFP4 weight format is the native release, not a lossy compression applied later.
MXFP4 is a 4-bit floating-point format with block-level scaling (the “MX” prefix refers to the microscaling approach), designed to preserve more dynamic range than a naive 4-bit integer quantization would. MXFP8 activations use 8-bit precision during the forward pass, which is a middle ground that keeps numerical stability better than running everything at 4-bit.
Practically, this means the weights you download are already compact relative to the parameter count. A back-of-envelope estimate: 2.8 trillion parameters at roughly 4 bits each is about 1.4 terabytes, before adding the vision encoder, embedding tables, attention projections that may be stored at slightly higher precision, and any KV cache. That’s still an enormous amount of storage and memory, just smaller than the 5.6 terabytes-plus you’d expect from an FP16 release of a model this size.
What hardware do you actually need? #
Enterprises have Remy.
⌂ Self-hosted
✓ SOC 2
✚ HIPAA
☰ Audit log
There’s no getting around it: this is not a single-GPU or even single-node consumer setup. At roughly 1.4TB for the base weights alone, you need enough aggregate GPU memory (or a combination of GPU and fast CPU/NVMe offload) across multiple accelerators to hold the model, plus headroom for KV cache that grows with context length and batch size. Given the 1M-token context support, anyone running long-context workloads needs to budget significant additional memory beyond the static weight footprint, particularly because attention state scales with sequence length even with the more efficient KDA and Gated DSA layers.
In practice, this puts K3.1 Pro in the territory of multi-GPU server racks: setups with eight or more high-memory data center GPUs networked with high-bandwidth interconnects, the kind of infrastructure used for serving large MoE models in production rather than for a home lab. MoE architectures do help here because inference only needs to route through 104B activated parameters per token, which keeps compute (as opposed to memory) more tractable than a dense model of the same total size. But you still need the full 2.8T parameters resident in memory somewhere, because any token could route to any expert.
Are there lower-bit options for smaller setups? #
Yes. The community has already produced quantizations below the native MXFP4 release. One example is an EXL3 format build at 2-bit precision, distributed in sharded safetensors files using the Trellis quantization library. Going from 4-bit to 2-bit roughly halves the memory footprint again, which can be the difference between a workload that needs an eight-GPU node and one that might run on four cards, depending on context length and batching.
The tradeoff is accuracy. Quantization-aware training at MXFP4 means Moonshot optimized the model’s weights specifically for that precision, so the 4-bit release should retain most of the quality of the original training run. A further post-hoc compression to 2-bit, done without retraining, generally introduces more degradation, particularly on tasks requiring precise reasoning or code generation where K3.1 Pro’s benchmark scores (strong results on Terminal-Bench, SWE Marathon, and various coding and agentic evaluations) were measured at full MXFP4 precision, not at a further-compressed format.
Is self-hosting Kimi K3.1 Pro worth it? #
For most individual developers and small teams, the honest answer is no. The hardware bar (terabyte-scale memory across multiple accelerators, fast interconnects, and the operational overhead of running MoE inference infrastructure) puts this well outside typical self-hosting budgets. This is the kind of model built for organizations that already run GPU clusters, want full control over data residency, or need to fine-tune or modify the model in ways the hosted API doesn’t allow.
For everyone else, the [Kimi API or chat interface](https://goremy.ai/blog/articles/how-to-run-ai-agents-locally-a-practical-setup-guide) delivers the same model without the infrastructure burden. The open-weight release matters less as a “run it at home” proposition and more as a transparency and research artifact: it lets the community inspect the architecture, build alternative quantizations, and verify claims about the model’s training and behavior, even if few people actually run the full thing locally.
If you do have access to a multi-GPU cluster and want to experiment, starting with the native MXFP4 release preserves the most fidelity to Moonshot’s reported benchmark numbers. Dropping to a community 2-bit quantization is a reasonable move only if you’re memory-constrained and can tolerate some quality loss, and you should validate behavior on your specific use case rather than assuming benchmark scores carry over unchanged.
Frequently Asked Questions #
How much VRAM does Kimi K3.1 Pro need to run locally?
mission-criticalenterprise applications.
● Live
-
Propellant load**✓ Go**
-
Range safety**✓ Go**
-
Weather**✓ Go**
-
Final pollT–00:42 At its native MXFP4 precision, the roughly 2.8 trillion parameters require on the order of 1.4 terabytes just for weight storage, before adding KV cache for context and other overhead. This requires multiple high-memory GPUs working together, not a single card.
What is MXFP4 and why does it matter here?
MXFP4 is a 4-bit floating-point format with block-level scaling that Moonshot used during quantization-aware training, meaning the model was trained with this precision in mind rather than quantized afterward. It gives a smaller, more efficient native release than a full-precision checkpoint would.
Can I run Kimi K3.1 Pro on a single GPU?
No. Even at 4-bit precision the weight footprint is far beyond what any single consumer or data center GPU offers in memory. Running it requires a multi-GPU setup with enough combined memory to hold the full parameter set plus context overhead.
Are there smaller quantized versions available?
Yes, community quantizations exist below the native MXFP4 format, including a 2-bit EXL3 version that further reduces memory requirements at some cost to output quality, since it wasn’t produced through the original quantization-aware training process.
Is it easier to just use the Kimi API instead of self-hosting?
For the overwhelming majority of developers, yes. The hardware required to self-host K3.1 Pro is server-cluster scale, so using Moonshot’s hosted API or chat product avoids the capital and operational cost of running the model yourself.