{"slug": "kimi-k3-run-on-rtx-5090", "title": "Kimi k3 run on RTX 5090", "summary": "RunInfra enables running Kimi-Linear-48B, a distilled version of the full 2.78-trillion-parameter Kimi K3 model, on a single consumer GPU such as the RTX 5090 with 32 GB VRAM, achieving 113.83 tokens per second at 3.18x speedup over fused INT4 kernels. The company also offers the full Kimi K3 as a deployable package on an 8x B300 node for production use, and provides a standalone adapter called k3 that translates between OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages protocols so coding agents like Claude Code, Codex, Aider, Cline, and opencode can connect to the local model unchanged.", "body_md": "Sketch is illustrative. This repository serves Kimi-Linear-48B, measured on an NVIDIA L40S inside a hard 32 GiB cap. The full Kimi K3 is 2.78T parameters and runs on an 8x B300 node, linked below.\n\nTL;DR.Run Kimi on your own consumer GPU, not someone's API. A single 32 GB card (RTX 5090, or any datacenter card) serves Kimi-Linear-48B at113.83 tok/s, up3.18xfrom fused INT4 kernels, and Claude Code, Codex, Cline, Aider and opencode all connect to it unchanged.\n\nNeed the full Kimi K3 in production?2.8 trillion parameters will not fit a workstation. If you want it running inside your own network instead of behind a closed-source API, RunInfra ships it as a deployable package with a pinned vLLM build, a benchmark receipt and weight verification:[Kimi K3 on 8x B300 →]\n\nPoint any coding agent at a local Kimi model and it works: `k3`\n\ntranslates between the agent's protocol and the model's.\n\n``` php\nflowchart LR\n    CC[\"Claude Code\"] -- \"Anthropic Messages\" --> K3\n    CX[\"Codex\"] -- \"OpenAI Responses\" --> K3\n    OT[\"Aider, Cline, opencode\"] -- \"OpenAI Chat\" --> K3\n\n    K3{{\"k3<br/>detects the client per request<br/>translates all three dialects<br/>tool calls: hermes, json, kimi,<br/>kimi_k3, pythonic, passthrough<br/>restores reasoning across turns\"}}\n\n    K3 -- \"OpenAI Chat\" --> EN[\"local engine<br/>Kimi-Linear-48B<br/>selective INT4<br/>fused decode kernels\"]\n\n    classDef client fill:#0d1117,stroke:#30363d,color:#c9d1d9\n    classDef bridge fill:#1f6feb,stroke:#388bfd,color:#ffffff\n    classDef engine fill:#238636,stroke:#2ea043,color:#ffffff\n    class CC,CX,OT client\n    class K3 bridge\n    class EN engine\n```\n\nLocal model servers usually speak OpenAI Chat Completions. Claude Code speaks Anthropic Messages. Codex speaks OpenAI Responses. Point the wrong client at the wrong endpoint and the request fails. `k3`\n\nsits between them, detects the caller on each request, and translates the request and response.\n\nThe protocol landscape has changed. Current releases of [vLLM](https://docs.vllm.ai/en/stable/serving/online_serving/), [llama.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), and [Ollama](https://docs.ollama.com/api/openai-compatibility) now document all three protocol families. `k3`\n\nis different because it is a standalone adapter for an existing OpenAI Chat Completions backend.\n\n| Capability | vLLM | llama.cpp | Ollama | `k3` |\n|---|---|---|---|---|\n| OpenAI Chat Completions | Documented | Documented | Documented | Verified |\n| OpenAI Responses | Documented | Documented | Documented, non-stateful | Verified |\n| Anthropic Messages | Documented | Documented | Documented | Verified |\n| Serves model inference itself | Yes | Yes | Yes | No, proxy only |\n| Detects a client preset from route, headers, user agent, and body hints | Not documented | Not documented | Not documented | Verified |\n| Translates tool calls to and from a separate OpenAI Chat upstream | Not applicable | Not applicable | Not applicable | Verified |\n| Ledger-backed reasoning restoration across client turns | Not documented | Not documented | Not documented | Verified when the backend supplies reasoning |\n\nThe vendor columns describe their public server documentation checked on July 29, 2026. \"Not documented\" is not a claim that the behavior is impossible. Ollama's [Anthropic compatibility](https://docs.ollama.com/api/anthropic-compatibility) is documented separately. The `k3`\n\ncolumn is verified against [ k3/dialects/](/RightNow-AI/local-kimi/blob/main/k3/dialects),\n\n[,](/RightNow-AI/local-kimi/blob/main/k3/detect.py)\n\n`k3/detect.py`\n\n[, and](/RightNow-AI/local-kimi/blob/main/k3/presets.py)\n\n`k3/presets.py`\n\n[.](/RightNow-AI/local-kimi/blob/main/k3/reasoning.py)\n\n`k3/reasoning.py`\n\nThis path assumes an OpenAI-compatible Kimi server is already listening at `http://127.0.0.1:8000/v1`\n\n.\n\n```\ngit clone https://github.com/RightNow-AI/local-kimi\ncd local-kimi\nuv sync --frozen\nuv run k3 serve --upstream http://127.0.0.1:8000/v1 --model kimi-linear --reasoning-field inline\n```\n\nIn another terminal, from the project Claude Code should work on:\n\n```\nexport ANTHROPIC_BASE_URL=http://localhost:8080\nexport ANTHROPIC_AUTH_TOKEN=local\nclaude\n```\n\nThe serve flags above are defined in [ k3/cli.py](/RightNow-AI/local-kimi/blob/main/k3/cli.py), and\n\n[runs this exact command as a subprocess on every CI run, sends the request Claude Code sends, and asserts the upstream was called in OpenAI Chat Completions. See the](/RightNow-AI/local-kimi/blob/main/tests/test_quickstart_path.py)\n\n`tests/test_quickstart_path.py`\n\n[full quickstart](/RightNow-AI/local-kimi/blob/main/docs/QUICKSTART.md)for llama.cpp setup, the model download, and troubleshooting, which are the parts this repository does not own.\n\n`k3`\n\nadded**0.231 ms per request** in a local CPU-only ASGI benchmark that sent the same request directly to a stub backend and through the proxy. The benchmark used no socket or network. The host hardware was not recorded, so this number describes that run rather than a portable latency guarantee.- The selective INT4 artifact is\n**28,803,304,448 bytes**, which is** 3.41x smaller**than the** 98,245,528,576-byte BF16 source tensor storage**. It was built and verified from the real checkpoint on one NVIDIA H100. See.`engine/quant/QUANTIZATION-RESULTS.md`\n\n- The engine decodes at\n**113.83 tokens per second** on one NVIDIA L40S, up from**35.76** before the fused kernels, a**3.18x** gain. The run used one stream, a 17-token prompt, 64 generated tokens, greedy decoding, five repeats, and a hard 32 GiB process cap. Peak reserved memory fell from 29.56 GiB to 27.63 GiB. Seefor the kernel-by-kernel breakdown and`engine/kernels/RESULTS.md`\n\nfor the profile that found the bottleneck.`engine/klinear/DECODE-PROFILE.md`\n\n- Two W4A16 kernels held\n**77% of decode time** at**8 to 10 percent** of the card's memory bandwidth, because both had been written as matrix-matrix products and were being used at decode as matrix-vector products. Rewriting them as real GEMVs is the entire gain.\n\nEvery measurement here names the hardware it ran on and ships the runner that\nproduced it. The experiments that did not work are recorded too, including three\nkernel fusions that made things slower and a head-to-head against llama.cpp:\n[ engine/kernels/RESULTS.md](/RightNow-AI/local-kimi/blob/main/engine/kernels/RESULTS.md),\n\n[,](/RightNow-AI/local-kimi/blob/main/engine/OPTIMIZATION-LIMITS.md)\n\n`engine/OPTIMIZATION-LIMITS.md`\n\n[and](/RightNow-AI/local-kimi/blob/main/engine/VERSUS-LLAMACPP.md)\n\n`engine/VERSUS-LLAMACPP.md`\n\n[.](/RightNow-AI/local-kimi/blob/main/engine/accuracy/RESULTS.md)\n\n`engine/accuracy/RESULTS.md`\n\n**32 GB of VRAM or more.** The INT4 artifact holds 26.83 GiB of weights and\nneeds roughly 2.5 GiB beyond that for activations, state and the CUDA context.\nAn RTX 5090, L40S, A100 or H100 all qualify, and the published throughput is an\nL40S.\n\nFitting a 24 GB card needs the expert weights near three bits. That work is\nunder way and tracked in [ engine/CONSUMER-GPU.md](/RightNow-AI/local-kimi/blob/main/engine/CONSUMER-GPU.md).\n\nFor people who want the technical detail. Everything below is measured on one NVIDIA L40S under a hard 32 GiB cap, and every figure names the document it came from.\n\nWe profiled decode before changing anything. The result was not what we expected. CUDA graph capture had already removed launch overhead, so the 2,236 elementwise kernel launches per token cost only 3.4 ms between them. Two W4A16 kernels held 77 percent of the time.\n\nBoth of those kernels had been written as matrix-matrix multiplies and were being\nused at decode as matrix-vector multiplies. They called `tl.dot`\n\nwith a 16-row\naccumulator to multiply a single token, throwing away fifteen sixteenths of the\nwork, and they indexed packed weights with the output dimension on the fastest\naxis, so neighbouring threads pulled separate cache lines. Together that left\nthem running at 4 to 10 percent of the card's memory bandwidth.\n\nRewriting them as real GEMVs, with the reduction axis contiguous and a single-row accumulator, took the grouped expert kernel from 8.3 to 51.7 percent of peak on the kernel benchmark. Inside a full decode step the same kernel reaches 36 percent, because it contends with everything else; both numbers are measured and they measure different things.\n\nThe fused path also allocates no per-call partial buffers, so peak memory fell while throughput rose.\n\nKernels live behind a registry in [ engine/kernels/registry.py](/RightNow-AI/local-kimi/blob/main/engine/kernels/registry.py).\nEach operation has one reference implementation and any number of variants, and\n\n[compares every variant against the reference. That is how a kernel gets swapped in without guessing whether it changed the model.](/RightNow-AI/local-kimi/blob/main/engine/kernels/equivalence.py)\n\n`engine/kernels/equivalence.py`\n\n**The fast kernels are on by default.** Nothing needs configuring to get the\nnumber above. To go back to the reference path, or to try a variant that is\nregistered but switched off, set `KIMI_KERNELS`\n\n:\n\n```\nKIMI_KERNELS=w4a16_grouped=reference,w4a16_dense=reference   # the slow path\nKIMI_KERNELS=w4a16_swiglu=fused                              # a fusion that measured slower\n```\n\nThis ordering was a bug at first. The variants were registered but nothing\nselected them, so an ordinary run silently got the reference path while the\nbenchmarks, which select variants explicitly, reported the fast one.\n`tests/test_kernel_defaults.py`\n\nnow pins the shipped default so that cannot\nhappen again.\n\nFull numbers, including what we built and chose not to ship, are in\n[ engine/kernels/RESULTS.md](/RightNow-AI/local-kimi/blob/main/engine/kernels/RESULTS.md).\n\nEvery figure above is generated from the measured numbers by\n[ scripts/make_figures.py](/RightNow-AI/local-kimi/blob/main/scripts/make_figures.py) and is written twice, as PNG\nand as vector PDF in\n\n[for use in a paper. Set](/RightNow-AI/local-kimi/blob/main/docs/figures)\n\n`docs/figures/`\n\n`USE_TEX=1`\n\nto render through a real LaTeX toolchain if one is installed;\nwithout it the figures use Computer Modern through matplotlib's mathtext, which\nmatches.\n\n```\nuv run python scripts/make_figures.py\n```\n\nThis project is built and maintained by [RunInfra](https://runinfra.ai/).\n\nThis repository is the open half of the work: one model, one GPU, every number measured in public.\n\nThe other half is [ K3Turbo](https://runinfra.ai/catalog/kimi-k3-b300x8-k3turbo),\na commercial package that runs the\n\n**full 2.8 trillion parameter Kimi K3** on hardware you own. No API, no per-token billing, no prompts leaving your network. It is closed source and it ships as a deployable unit rather than a research repository:\n\n| Engine | A pinned vLLM build, compiled from source, with the serving configuration already tuned |\n| Proof | A benchmark receipt with measured latency, not a marketing figure |\n| Verification | A weight verifier that checks what you deploy against a pinned commit digest, so the bytes you run are the bytes that were measured |\n| Deployment | Docker Compose, Kubernetes, RunPod and Modal recipes |\n| Weights | Fetched separately from Hugging Face, never redistributed |\n\n**Be clear about the hardware.** Kimi K3 is 2.8 trillion parameters. K3Turbo\ntargets an **8x B300 node**, 288 GB per GPU and 2304 GB per node in a single\nNVLink domain. It does not fit an 8x B200 node, and nothing about it runs on a\nconsumer card. The published measurements are text-only on coding prompts of 34\nto 36 thousand tokens; the vision tower is uncharacterised.\n\nIf you want the largest open model in the world running inside your own walls, that is what it is for. If you want a 48B model on a single GPU, that is this repository, and it is free.\n\n| Path | Contents |\n|---|---|\n`k3/` |\nProtocol detection, translation, streaming, tool calls, and reasoning restoration |\n`docs/` |\nSetup guides for Claude Code, Codex, the OpenAI SDK, and the adapter architecture |\n`engine/` |\nKimi and Kimi-Linear experiments, measurements, and reference serving work |\n`tests/` |\nProtocol, regression, conformance, and overhead checks |\n`research/` |\nExploratory scripts and recorded conclusions |\n`scripts/` |\nRepository helper scripts |\n`reference/` |\nThird-party Moonshot material used as a test oracle under its upstream terms |\n`assets/` |\nImages used by the documentation |\n\nSee [LICENSE](/RightNow-AI/local-kimi/blob/main/LICENSE) and [LICENCE-DECISION.md](/RightNow-AI/local-kimi/blob/main/LICENCE-DECISION.md). Third-party files in `reference/`\n\nkeep their upstream terms.", "url": "https://wpnews.pro/news/kimi-k3-run-on-rtx-5090", "canonical_source": "https://github.com/RightNow-AI/local-kimi", "published_at": "2026-07-30 13:05:09+00:00", "updated_at": "2026-07-30 13:22:43.792744+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["RunInfra", "Kimi K3", "Kimi-Linear-48B", "NVIDIA RTX 5090", "NVIDIA L40S", "NVIDIA B300", "vLLM", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/kimi-k3-run-on-rtx-5090", "markdown": "https://wpnews.pro/news/kimi-k3-run-on-rtx-5090.md", "text": "https://wpnews.pro/news/kimi-k3-run-on-rtx-5090.txt", "jsonld": "https://wpnews.pro/news/kimi-k3-run-on-rtx-5090.jsonld"}}