Introduction #
On July 27, 2026, Moonshot AI released the model weights for Kimi-K3. Ever since the model was announced on July 16, it had generated significant buzz as “the largest open-weight model in history.”
At Fixstars, we deployed the model to a single-node NVIDIA B300 x8 environment on the very day the weights were released and ran an inference performance benchmark. This article is our first-look report. From the start of the download through the completion of the benchmark, we managed to run through the whole process on release day.
What Is Kimi-K3? #
Kimi-K3 is a frontier-class MoE (Mixture of Experts) model developed by Moonshot AI. The main specifications published in its model card are as follows.
| Item | Specification |
|---|---|
| Total parameters | 2.8T (2.8 trillion) |
| Active parameters | 104B |
| Expert configuration | 16 of 896 experts activated per token (+ 2 shared experts) |
| Number of layers | 93 (69 KDA layers + 24 Gated MLA layers) |
| Precision of released weights | MXFP4 weights / MXFP8 activations (quantization-aware training) |
| Context length | 1M tokens (1,048,576) |
| Multimodal | Native visual understanding supported (MoonViT-V2, 401M) |
| Reasoning mode | Thinking mode always on (intensity configurable: low / high / max) |
| License | Kimi K3 License (custom license) |
On the architecture side, Kimi-K3 adopts two proprietary techniques: Kimi Delta Attention (KDA), a hybrid linear attention, and Attention Residuals, a replacement for residual connections. The majority of the attention layers (69 of 93) are KDA, with only 24 conventional (Gated MLA) attention layers. As discussed later, this hybrid configuration directly affects memory management on the inference server.
Another important point is that the model is distributed as MXFP4 weights that underwent quantization-aware training (QAT) starting from the SFT stage. Rather than being quantized after training, it was designed from the outset to be used in MXFP4, so it can be deployed as-is without any additional quantization work.
In the various evaluations published at announcement, Kimi-K3 is rated as delivering the highest performance ever for an open-weight model, coming close to top-tier proprietary models such as Claude Fable 5 and GPT 5.6 Sol. Its strength on coding and agent tasks has drawn particular attention.
Test Environment #
| Item | Details |
|---|---|
| GPU | NVIDIA B300 SXM6 x8 (single node) |
| GPU memory | 288GB HBM3e x8 = 2,304GB total |
| CPU | Intel Xeon 6767P × 2 sockets (64 cores/socket, 128 cores total, 1 thread/core) |
| System memory | 3.0 TiB DDR5 |
| Interconnect | NVLink 5.0, 18 links/GPU, 53.125 GB/s/link = 956 GB/s/GPU (unidirectional) |
| OS / Driver | Ubuntu 24.04.4 LTS (kernel 6.8.0-71-generic), CUDA 13.0, Driver 580.105.08 |
| Inference engine | SGLang 0.5.16 (Docker container sglang-kimi-k3) |
Why SGLang? #
At release, both vLLM and SGLang supported Kimi-K3, but we chose SGLang. The reason is its support for DCP (Decode Context Parallelism).
For a hybrid model like Kimi-K3, the memory efficiency of the KV cache determines the effective context length. At release, vLLM did not support DCP, so the KV cache is replicated across all GPUs. SGLang, on the other hand, supports DCP and can distribute the MLA KV cache across 8 GPUs, allowing it to achieve 8x the effective context capacity with the same amount of memory.
Considering the Memory Footprint #
A model with 2.8T total parameters would require roughly 2.8TB just for the weights even in FP8, which would not fit within this environment’s total GPU memory (about 2.2TB). However, Kimi-K3’s released weights are provided in MXFP4, and in our measurements the weights came to about 196GB per GPU, or about 1.57TB across 8 GPUs. This works out to fit as-is on a single B300 x8 node — a design that, despite the model’s enormous scale, was clearly built with running “on a single node” in mind. The measured memory breakdown is described later.
Deployment #
Down the Weights
The MXFP4 weights total about 1.6TB. The download completed in about 1 hour. Partway through, some files got stuck down; killing the process and rerunning it completed without issue. Access is heavily concentrated right after release, so if you run into the same phenomenon, we recommend retrying.
Launch Command
docker run --gpus all \
--shm-size 32g \
--network host \
-v /data/hf_cache:/root/.cache/huggingface \
--ipc=host \
lmsysorg/sglang:kimi-k3 \
sglang serve \
--trust-remote-code \
--model-path moonshotai/Kimi-K3 \
--tp-size 8 \
--dcp-size 8 \
--disable-custom-all-reduce \
--mem-fraction-static 0.85 \
--reasoning-parser kimi_k3 \
--tool-call-parser kimi_k3 \
--mamba-full-memory-ratio 5 \
--speculative-algorithm DSPARK \
--speculative-draft-model-path RadixArk/Kimi-K3-DSpark \
--speculative-dspark-block-size 7 \
--enable-linear-replayssm-spec \
--host 0.0.0.0 \
--port 8001
A few notes on the key options:
--mamba-full-memory-ratio
sets the memory allocation ratio between the KDA state pool and the MLA KV pool. A larger value grows the KDA state pool (suited to short-text workloads with many concurrent requests), while a smaller value grows the MLA KV pool (suited to long-text workloads). You can compute it by entering the average request length into the Mamba ratio calculator in the SGLang cookbook. This time we first launched with 5, which — as noted below — became a point of reflection.
--speculative-algorithm DSPARK
configures speculative decoding. It is combined with the publicly available draft model RadixArk/Kimi-K3-DSpark.
Launch Time: About 89 Minutes Total
The total time to complete launch was about 88.8 minutes, of which model took about 81.4 minutes. a 1.6TB-class set of weights makes this unavoidable, but taking an hour and a half every time you restart with a changed setting will really add up during future tuning work.
One gotcha: docker logs
does not show progress during model load, and it goes completely silent for over 20 minutes. It’s unnerving — you wonder if it has hung — but docker logs -f
or docker attach
does let you see the progress display.
GPU Memory Breakdown and KV Cache
Here is the breakdown of GPU memory usage in steady state after launch. The figures are per-rank (i.e., for a single GPU), taken from rank TP0 as a representative example; “× DCP8” below means a value is distributed across the 8 GPUs of the Decode Context Parallelism group, so the whole-cluster total is 8× the per-rank figure.
| Item | Size (GB) | Notes |
|---|---|---|
| Total GPU memory | 275.04 | B300 SXM6 |
| In use (nvidia-smi) | 248.02 | steady state after launch |
| Free | 26.09 | |
| └ Model weights (main) | 195.86 | MXFP4 (compressed-tensors) |
| └ Model weights (DSPARK draft) | 0.56 | |
| └ Mamba Cache (KDA state pool) | 17.13 | conv 0.56 + ssm 15.87 + conv_window 0.70 |
| └ MLA KV Cache | 1.70 | 65,984 tokens (per-rank, bf16) |
| └ KDA KV (K+V) | 1.26 | 527,872 tokens (per-rank, bf16) |
| └ Other (NCCL / CUDA context / MM IPC, etc.) | ~31.5 |
Since Kimi-K3 is an MLA + KDA hybrid model, three pools equivalent to the KV cache are allocated.
| Pool | Per-rank tokens | Total | Purpose |
|---|---|---|---|
| MLA KV | 65,984 | 527,872 (× DCP8) | Multi-Head Latent Attention. The primary metric for context length |
| KDA KV (K+V) | 527,872 | 527,872 (replica) | Kimi Delta Attention |
| Mamba state | 313 requests | 313 | Concurrent-request cap for KDA |
The effective context capacity is 527,872 tokens (MLA KV per-rank 65,984 × DCP8).
Reflection: --mamba-full-memory-ratio=5
Was Too Large
We realized after launch that the default value of --mamba-full-memory-ratio
is 0.9. A value of 5 leans too heavily toward the KDA state pool, and as a result the KV cache’s token capacity was limited to about 0.5M. For coding-agent use cases, context tends to get long, so this value should have been made much smaller to allocate more memory to the KV cache. The right approach is to compute it with the cookbook calculator to match your workload.
Benchmark Results #
We measured throughput and latency under two types of load. The metric abbreviations used in the tables are: ISL = input sequence length, OSL = output sequence length, out_tput = output throughput, total_tput = total (input + output) throughput, TTFT = time to first token, ITL = inter-token latency, and med/p90/p99 = median / 90th-percentile / 99th-percentile.
Random (ISL=8K, OSL=1K, num-prompts 200)
| Concurrency | duration (s) | out_tput (tok/s) | total_tput (tok/s) | TTFT med/p90/p99 (ms) | ITL med/p90/p99 (ms) | failed |
|---|---|---|---|---|---|---|
| 10 | 543.4 | 376.9 | 3,391.9 | 624 / 1,137 / 5,180 | 37.9 / 39 / 509 | 0 |
| 20 | 397.0 | 515.8 | 4,642.6 | 675 / 1,902 / 10,772 | 50.9 / 52 / 654 | 0 |
| 30 | 338.3 | 605.4 | 5,448.2 | 746 / 6,250 / 16,784 | 60.4 / 63 / 797 | 0 |
| 40 | 315.2 | 649.7 | 5,847.6 | 1,170 / 12,368 / 22,238 | 70.5 / 76 / 825 | 0 |
| 50 | 225.9 | 367.2 | 3,305.1 | 10,934 / 46,651 / 55,621 | 71.7 / 76 / 810 | 119 |
Each row processes 1,638,400 input tokens and 204,800 output tokens (the concurrency-50 row is lower — 663,552 input and 82,944 output — because of the failed requests).
Internal Real-World Coding-Agent Dataset (max-duration 300, multi-turn)
| Concurrency | duration (s) | out_tput (tok/s) | total_tput (tok/s) | TTFT med/p90/p99 (ms) | ITL med/p90/p99 (ms) | failed |
|---|---|---|---|---|---|---|
| 10 | 396.4 | 481.6 | 2,477.6 | 370 / 670 / 731 | 38.7 / 39 / 87 | 0 |
| 20 | 405.4 | 672.0 | 3,787.7 | 393 / 986 / 2,469 | 51.7 / 54 / 341 | 0 |
| 30 | 458.4 | 754.2 | 3,879.6 | 409 / 1,422 / 2,410 | 61.1 / 62 / 359 | 0 |
| 40 | 686.8 | 629.4 | 2,925.8 | 452 / 2,073 / 3,400 | 70.6 / 73 / 392 | 0 |
| 50 | 726.6 | 791.9 | 3,276.7 | 4,454 / 9,135 / 14,245 | 74.6 / 76 / 458 | 0 |
Throughput Scaling
Figure — Kimi-K3 on NVIDIA B300 x8: throughput vs. concurrency.The left chart shows total throughput (input + output) and the right chart shows output throughput, both plotted against the number of concurrent requests.
As the figure shows, the Random workload plateaus around concurrency 40, and the real-world dataset around concurrency 30. At concurrency 50 on the Random workload, 119 of 200 requests failed and throughput collapsed.
Observations
Our candid impression: it’s fast. Given the 2.8T scale, we were braced for it to be slower, but it achieves speed on par with GLM-5.2 (in a B300 x4 configuration), which we have benchmarked previously. We attribute this to the combination of a highly sparse MoE with active parameters held to 104B, native MXFP4 weights, and speculative decoding via DSPARK.
Looking at the scaling characteristics, total throughput plateaus at concurrency 40 for the Random workload and at concurrency 30 for the real-world dataset. At concurrency 50 on the Random workload, the median TTFT exceeded 10 seconds, and 119 of 200 requests failed. On the real-world dataset as well, the degradation in ITL and TTFT becomes conspicuous beyond concurrency 40. With the current settings, it seems reasonable to view around 30 concurrent requests as the practical upper limit for simultaneous processing.
Coding Performance Assessment (Subjective) #
Beyond throughput, we also wanted to verify output quality, so as a task close to real-world work we had it implement an “A* pathfinding visualization tool” as a single HTML file. The prompt deliberately packed in requirements that tend to expose sloppy implementations: a hand-written binary heap, re-open handling when the g-score improves, explicit tie-breaking rules with the reasoning commented, a two-pane comparison mode, and so on.
(The original article includes the full prompt text at this point.)
Kimi-K3 implemented this prompt correctly in one shot in about 15 minutes, and it worked simply by opening it in a browser. For comparison, we gave the same prompt to GLM-5.2 (NVFP4) and put the two outputs side by side and used them. This is purely subjective, but we felt the appearance and usability were better with Kimi-K3.
Figure — Kimi-K3’s output.Two heuristics running simultaneously in comparison mode. The legend/controls panel and keyboard shortcuts are extra polish beyond the requirements.
Figure — GLM-5.2 (NVFP4)’s output.It largely satisfies the requirements, but a gap shows in the overall polish of layout and color scheme.
Here are a few things we noticed from reading the generated code:
Both hand-implemented the priority queue as a binary heap (a MinHeap class) as required, rather than cheating with an array sort.
For tie-breaking, Kimi-K3 used a three-tier scheme (“f equal → h ascending → g descending → LIFO”), explaining the reason for each tier in comments and also spelling out the rule in the UI legend. GLM-5.2 used only a single tier (h ascending) — also with a reasoning comment.
Kimi-K3 spontaneously added keyboard shortcuts not requested in the prompt (Space to play / → to step / V for comparison mode) and a legend/controls panel.
GLM-5.2 also had its own touches, such as a random-wall-generation button, but the overall polish including layout and color scheme was in Kimi-K3’s favor.
Since this is not a quantitative benchmark, take it as a reference only, but in terms of the perceived quality as an inference backend for a coding agent, it made a good impression alongside its speed.
Discussion #
The biggest achievement is that a 2.8T model ran at practical speeds on a single node. Thanks to native distribution in MXFP4 (with QAT already done), you can download it and launch it as-is with no additional quantization work, and it fits on a B300 x8. This is a distribution format that takes advantage of the FP4 compute performance of the B300 (Blackwell Ultra).
Operational challenges also came into view. Because launch (model ) takes about an hour and a half, you need to factor launch time into trial-and-error over settings and into failover design. In addition, because of the KDA + MLA hybrid configuration, there are model-specific memory-allocation parameters like --mamba-full-memory-ratio
, and tuning them to your workload (short-text high-concurrency vs. long-text agents) governs the real-world performance. Our settings this time leaned too heavily toward the KDA state pool, so if you’re targeting coding-agent use cases, the next step is to lower the ratio to expand the KV cache and verify behavior at long context.
Summary #
On the day Kimi-K3’s weights were released, we completed deployment to a single-node NVIDIA B300 x8 and ran benchmarks. Thanks to native MXFP4 distribution, the weights fit in about 1.6TB and can be launched on a single node with no additional quantization. Throughput scaled to roughly 30 concurrent requests, and for a 2.8T model the speed was practical, exceeding our expectations. On output quality as well, it handled the A* visualization implementation task in one shot and even showed thoughtful extra polish beyond the requirements.
This evaluation is more than just a first look at a new model — it ties directly into our own products.
Fixstars offers “Fixstars Vega,” an on-premises AI appliance that combines open-weight LLMs with our proprietary performance-engineering harness. In Vega, we continuously evaluate the latest open-weight models, and this Kimi-K3 evaluation is part of that effort.
From the day a new model is released, you can safely put the latest open-weight models to use in a secure local environment. That is the vision Vega aims for. If you’re interested in applying AI to embedded software development, please take a look at the product page.
Reference Links #
Kimi-K3(Hugging Face)Kimi K3 Tech Blog(Moonshot AI)Kimi K3 Technical ReportSGLang cookbook: Kimi-K3