Kimi k3 run on RTX 5090

RunInfra enables running Kimi-Linear-48B, a distilled version of the full 2.78-trillion-parameter Kimi K3 model, on a single consumer GPU such as the RTX 5090 with 32 GB VRAM, achieving 113.83 tokens per second at 3.18x speedup over fused INT4 kernels. The company also offers the full Kimi K3 as a deployable package on an 8x B300 node for production use, and provides a standalone adapter called k3 that translates between OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages protocols so coding agents like Claude Code, Codex, Aider, Cline, and opencode can connect to the local model unchanged.

Sketch is illustrative. This repository serves Kimi-Linear-48B, measured on an NVIDIA L40S inside a hard 32 GiB cap. The full Kimi K3 is 2.78T parameters and runs on an 8x B300 node, linked below. TL;DR.Run Kimi on your own consumer GPU, not someone's API. A single 32 GB card RTX 5090, or any datacenter card serves Kimi-Linear-48B at113.83 tok/s, up3.18xfrom fused INT4 kernels, and Claude Code, Codex, Cline, Aider and opencode all connect to it unchanged. Need the full Kimi K3 in production?2.8 trillion parameters will not fit a workstation. If you want it running inside your own network instead of behind a closed-source API, RunInfra ships it as a deployable package with a pinned vLLM build, a benchmark receipt and weight verification: Kimi K3 on 8x B300 → Point any coding agent at a local Kimi model and it works: k3 translates between the agent's protocol and the model's. php flowchart LR CC "Claude Code" -- "Anthropic Messages" -- K3 CX "Codex" -- "OpenAI Responses" -- K3 OT "Aider, Cline, opencode" -- "OpenAI Chat" -- K3 K3{{"k3<br/ detects the client per request<br/ translates all three dialects<br/ tool calls: hermes, json, kimi,<br/ kimi k3, pythonic, passthrough<br/ restores reasoning across turns"}} K3 -- "OpenAI Chat" -- EN "local engine<br/ Kimi-Linear-48B<br/ selective INT4<br/ fused decode kernels" classDef client fill: 0d1117,stroke: 30363d,color: c9d1d9 classDef bridge fill: 1f6feb,stroke: 388bfd,color: ffffff classDef engine fill: 238636,stroke: 2ea043,color: ffffff class CC,CX,OT client class K3 bridge class EN engine Local model servers usually speak OpenAI Chat Completions. Claude Code speaks Anthropic Messages. Codex speaks OpenAI Responses. Point the wrong client at the wrong endpoint and the request fails. k3 sits between them, detects the caller on each request, and translates the request and response. The protocol landscape has changed. Current releases of vLLM https://docs.vllm.ai/en/stable/serving/online serving/ , llama.cpp https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md , and Ollama https://docs.ollama.com/api/openai-compatibility now document all three protocol families. k3 is different because it is a standalone adapter for an existing OpenAI Chat Completions backend. | Capability | vLLM | llama.cpp | Ollama | k3 | |---|---|---|---|---| | OpenAI Chat Completions | Documented | Documented | Documented | Verified | | OpenAI Responses | Documented | Documented | Documented, non-stateful | Verified | | Anthropic Messages | Documented | Documented | Documented | Verified | | Serves model inference itself | Yes | Yes | Yes | No, proxy only | | Detects a client preset from route, headers, user agent, and body hints | Not documented | Not documented | Not documented | Verified | | Translates tool calls to and from a separate OpenAI Chat upstream | Not applicable | Not applicable | Not applicable | Verified | | Ledger-backed reasoning restoration across client turns | Not documented | Not documented | Not documented | Verified when the backend supplies reasoning | The vendor columns describe their public server documentation checked on July 29, 2026. "Not documented" is not a claim that the behavior is impossible. Ollama's Anthropic compatibility https://docs.ollama.com/api/anthropic-compatibility is documented separately. The k3 column is verified against k3/dialects/ /RightNow-AI/local-kimi/blob/main/k3/dialects , , /RightNow-AI/local-kimi/blob/main/k3/detect.py k3/detect.py , and /RightNow-AI/local-kimi/blob/main/k3/presets.py k3/presets.py . /RightNow-AI/local-kimi/blob/main/k3/reasoning.py k3/reasoning.py This path assumes an OpenAI-compatible Kimi server is already listening at http://127.0.0.1:8000/v1 . git clone https://github.com/RightNow-AI/local-kimi cd local-kimi uv sync --frozen uv run k3 serve --upstream http://127.0.0.1:8000/v1 --model kimi-linear --reasoning-field inline In another terminal, from the project Claude Code should work on: export ANTHROPIC BASE URL=http://localhost:8080 export ANTHROPIC AUTH TOKEN=local claude The serve flags above are defined in k3/cli.py /RightNow-AI/local-kimi/blob/main/k3/cli.py , and runs this exact command as a subprocess on every CI run, sends the request Claude Code sends, and asserts the upstream was called in OpenAI Chat Completions. See the /RightNow-AI/local-kimi/blob/main/tests/test quickstart path.py tests/test quickstart path.py full quickstart /RightNow-AI/local-kimi/blob/main/docs/QUICKSTART.md for llama.cpp setup, the model download, and troubleshooting, which are the parts this repository does not own. k3 added 0.231 ms per request in a local CPU-only ASGI benchmark that sent the same request directly to a stub backend and through the proxy. The benchmark used no socket or network. The host hardware was not recorded, so this number describes that run rather than a portable latency guarantee.- The selective INT4 artifact is 28,803,304,448 bytes , which is 3.41x smaller than the 98,245,528,576-byte BF16 source tensor storage . It was built and verified from the real checkpoint on one NVIDIA H100. See. engine/quant/QUANTIZATION-RESULTS.md - The engine decodes at 113.83 tokens per second on one NVIDIA L40S, up from 35.76 before the fused kernels, a 3.18x gain. The run used one stream, a 17-token prompt, 64 generated tokens, greedy decoding, five repeats, and a hard 32 GiB process cap. Peak reserved memory fell from 29.56 GiB to 27.63 GiB. Seefor the kernel-by-kernel breakdown and engine/kernels/RESULTS.md for the profile that found the bottleneck. engine/klinear/DECODE-PROFILE.md - Two W4A16 kernels held 77% of decode time at 8 to 10 percent of the card's memory bandwidth, because both had been written as matrix-matrix products and were being used at decode as matrix-vector products. Rewriting them as real GEMVs is the entire gain. Every measurement here names the hardware it ran on and ships the runner that produced it. The experiments that did not work are recorded too, including three kernel fusions that made things slower and a head-to-head against llama.cpp: engine/kernels/RESULTS.md /RightNow-AI/local-kimi/blob/main/engine/kernels/RESULTS.md , , /RightNow-AI/local-kimi/blob/main/engine/OPTIMIZATION-LIMITS.md engine/OPTIMIZATION-LIMITS.md and /RightNow-AI/local-kimi/blob/main/engine/VERSUS-LLAMACPP.md engine/VERSUS-LLAMACPP.md . /RightNow-AI/local-kimi/blob/main/engine/accuracy/RESULTS.md engine/accuracy/RESULTS.md 32 GB of VRAM or more. The INT4 artifact holds 26.83 GiB of weights and needs roughly 2.5 GiB beyond that for activations, state and the CUDA context. An RTX 5090, L40S, A100 or H100 all qualify, and the published throughput is an L40S. Fitting a 24 GB card needs the expert weights near three bits. That work is under way and tracked in engine/CONSUMER-GPU.md /RightNow-AI/local-kimi/blob/main/engine/CONSUMER-GPU.md . For people who want the technical detail. Everything below is measured on one NVIDIA L40S under a hard 32 GiB cap, and every figure names the document it came from. We profiled decode before changing anything. The result was not what we expected. CUDA graph capture had already removed launch overhead, so the 2,236 elementwise kernel launches per token cost only 3.4 ms between them. Two W4A16 kernels held 77 percent of the time. Both of those kernels had been written as matrix-matrix multiplies and were being used at decode as matrix-vector multiplies. They called tl.dot with a 16-row accumulator to multiply a single token, throwing away fifteen sixteenths of the work, and they indexed packed weights with the output dimension on the fastest axis, so neighbouring threads pulled separate cache lines. Together that left them running at 4 to 10 percent of the card's memory bandwidth. Rewriting them as real GEMVs, with the reduction axis contiguous and a single-row accumulator, took the grouped expert kernel from 8.3 to 51.7 percent of peak on the kernel benchmark. Inside a full decode step the same kernel reaches 36 percent, because it contends with everything else; both numbers are measured and they measure different things. The fused path also allocates no per-call partial buffers, so peak memory fell while throughput rose. Kernels live behind a registry in engine/kernels/registry.py /RightNow-AI/local-kimi/blob/main/engine/kernels/registry.py . Each operation has one reference implementation and any number of variants, and compares every variant against the reference. That is how a kernel gets swapped in without guessing whether it changed the model. /RightNow-AI/local-kimi/blob/main/engine/kernels/equivalence.py engine/kernels/equivalence.py The fast kernels are on by default. Nothing needs configuring to get the number above. To go back to the reference path, or to try a variant that is registered but switched off, set KIMI KERNELS : KIMI KERNELS=w4a16 grouped=reference,w4a16 dense=reference the slow path KIMI KERNELS=w4a16 swiglu=fused a fusion that measured slower This ordering was a bug at first. The variants were registered but nothing selected them, so an ordinary run silently got the reference path while the benchmarks, which select variants explicitly, reported the fast one. tests/test kernel defaults.py now pins the shipped default so that cannot happen again. Full numbers, including what we built and chose not to ship, are in engine/kernels/RESULTS.md /RightNow-AI/local-kimi/blob/main/engine/kernels/RESULTS.md . Every figure above is generated from the measured numbers by scripts/make figures.py /RightNow-AI/local-kimi/blob/main/scripts/make figures.py and is written twice, as PNG and as vector PDF in for use in a paper. Set /RightNow-AI/local-kimi/blob/main/docs/figures docs/figures/ USE TEX=1 to render through a real LaTeX toolchain if one is installed; without it the figures use Computer Modern through matplotlib's mathtext, which matches. uv run python scripts/make figures.py This project is built and maintained by RunInfra https://runinfra.ai/ . This repository is the open half of the work: one model, one GPU, every number measured in public. The other half is K3Turbo https://runinfra.ai/catalog/kimi-k3-b300x8-k3turbo , a commercial package that runs the full 2.8 trillion parameter Kimi K3 on hardware you own. No API, no per-token billing, no prompts leaving your network. It is closed source and it ships as a deployable unit rather than a research repository: | Engine | A pinned vLLM build, compiled from source, with the serving configuration already tuned | | Proof | A benchmark receipt with measured latency, not a marketing figure | | Verification | A weight verifier that checks what you deploy against a pinned commit digest, so the bytes you run are the bytes that were measured | | Deployment | Docker Compose, Kubernetes, RunPod and Modal recipes | | Weights | Fetched separately from Hugging Face, never redistributed | Be clear about the hardware. Kimi K3 is 2.8 trillion parameters. K3Turbo targets an 8x B300 node , 288 GB per GPU and 2304 GB per node in a single NVLink domain. It does not fit an 8x B200 node, and nothing about it runs on a consumer card. The published measurements are text-only on coding prompts of 34 to 36 thousand tokens; the vision tower is uncharacterised. If you want the largest open model in the world running inside your own walls, that is what it is for. If you want a 48B model on a single GPU, that is this repository, and it is free. | Path | Contents | |---|---| k3/ | Protocol detection, translation, streaming, tool calls, and reasoning restoration | docs/ | Setup guides for Claude Code, Codex, the OpenAI SDK, and the adapter architecture | engine/ | Kimi and Kimi-Linear experiments, measurements, and reference serving work | tests/ | Protocol, regression, conformance, and overhead checks | research/ | Exploratory scripts and recorded conclusions | scripts/ | Repository helper scripts | reference/ | Third-party Moonshot material used as a test oracle under its upstream terms | assets/ | Images used by the documentation | See LICENSE /RightNow-AI/local-kimi/blob/main/LICENSE and LICENCE-DECISION.md /RightNow-AI/local-kimi/blob/main/LICENCE-DECISION.md . Third-party files in reference/ keep their upstream terms.