{"slug": "benchmarking-kimi-k3-across-vllm-sglang-and-atom-on-mi350x", "title": "Benchmarking Kimi-K3 Across vLLM, SGLang, and ATOM on MI350X", "summary": "Moonshot AI released the weights for Kimi-K3, a 2.8-trillion-parameter, 1M-context, natively-MXFP4 Mixture-of-Experts model, with AMD Instinct support available on day 0 across three serving frameworks: vLLM, SGLang, and ATOM. AMD benchmarked the 1.56 TB model on an 8× MI350X node using its open-source MAD (Model Automation and Dashboarding) harness, running a shared sweep of 8192 input and 1024 output tokens at concurrency 1·4·8·16·32·64·128 with tensor-parallel degree 8 (TP8) on the gfx950 generation (MI350X and MI355X). The recipes are gated with skip_gpu_arch: \"gfx942\" to prevent unvalidated runs on MI300X and MI325X.", "body_md": "# Benchmarking Kimi-K3 Across vLLM, SGLang, and ATOM on MI350X[#](#benchmarking-kimi-k3-across-vllm-sglang-and-atom-on-mi350x)\n\nMoonshot AI released the weights for **Kimi-K3**, a 2.8-trillion-parameter,\n1M-context, natively-MXFP4 Mixture-of-Experts (MoE) model, and AMD Instinct™ support\nwas available on day 0 across **three independent serving frameworks**: vLLM, SGLang,\nand ATOM. The recipes target the gfx950 generation (MI350X and MI355X). The\nmeasurements in this post were taken on an 8× MI350X node.\n\nDay-0 announcements usually establish that a model runs. This post covers the next step: benchmarking a 1.56 TB model across three engines on new gfx950 silicon kernels, and knowing what the resulting numbers do and do not support. Three engines mean three container images, three server launchers, three benchmark clients, and three result formats, and each of those is a chance for an unnoticed difference in workload to turn a comparison into an artifact.\n\nThe answer used here is **MAD** (Model Automation and Dashboarding), AMD’s open-source\nbenchmarking harness for AMD Instinct™ GPUs. MAD keeps a declarative *model registry*,\n`models.json`, in which one entry describes everything a workload needs, including the\nDockerfile to build, the script to run, the dataset or checkpoint to fetch, and the\nresults file to collect. Its companion runner, **madengine**, reads that entry and\nexecutes the whole pipeline: build the image, launch the server, drive the benchmark\nclient, and emit a normalized CSV. Adding Kimi-K3 on a new engine is therefore a\nreviewable change to a registry entry, a Dockerfile, and a YAML config, rather than a\nshell session that nobody else can replay. This post covers that flow, the day-0\nresults it produced, and the limits of what those results establish.\n\nBy the end of this blog, you will know how to run the same day-0 Kimi-K3 benchmark on all three engines with one command each, how to read the CSV they share, how to retarget the sweep to your own input and output lengths and concurrency, and where the measurement’s boundaries lie.\n\n## Key Takeaways[#](#key-takeaways)\n\nThis post’s main findings, before the detail:\n\n- **One command per engine.**`madengine run --tags pyt_vllm_kimi-k3` (or`pyt_sglang_kimi-k3` ,`pyt_atom_kimi-k3` ) builds the image, launches the server, drives the benchmark, and\nemits a normalized`perf_Kimi-K3.csv` , with no manual container plumbing.\n- **One shared sweep, three engines.** All three frameworks run the same primary workload\naxes: 8192 input and 1024 output tokens, concurrency`1·4·8·16·32·64·128` , and a\ntensor-parallel degree of 8 (TP8), using each engine’s out-of-box configuration.\nSecondary settings still differ (model dtype, KV-cache dtype, prompt-length sampling,\nEOS handling). See Table 5.\n- **Declarative configs, not shell scripts.** Every server flag, environment variable,\nand sweep axis lives in a versioned YAML. The recipe is the config: reproducing a run\nmeans re-running the file.\n- **Your sweep, same harness.** Copy the config, set your own input sequence length (ISL)\nand output sequence length (OSL) and concurrency, and point a run at it with`--additional-context` . Same containers, same CSV schema.\n- **Hardware-aware guardrails.**`skip_gpu_arch: \"gfx942\"` keeps today’s gfx950 recipe\nfrom silently running on MI300X and MI325X, which it has not been validated for.`arch_overrides` is the general mechanism the shipped configs use to vary settings by\nGPU architecture.\n- **Consistency by design.** Automatic server-health gating, unbuffered logging, model-cache\nreuse, and a common core CSV schema help a run on your cluster line up with the runs in\nthis post.\n- **Fully disclosed methodology.** The workload definition, the verbatim server and\nbenchmark-client commands for all three engines, the server-side settings side by side,\nand the metric definitions are all stated in[Benchmark Methodology](#benchmark-methodology) . See Tables 8 and 9.\n- **An out-of-box snapshot.** A default`madengine run` on 8× MI350X shows the three\nengines converging to within 0.5% of each other at concurrency 32, with a wider spread\nboth below and above that point. Read it as a launch-day picture of three day-0\nout-of-box configurations, not as an engine comparison. See Figure 3.\n\n## Test Configuration and Day-0 Evidence[#](#test-configuration-and-day-0-evidence)\n\nThis section documents exactly what ran: the enablement timeline, the hardware, and the pinned software versions.\n\n### Day-0 Enablement, Per Engine[#](#day-0-enablement-per-engine)\n\nKimi-K3 weights were published by Moonshot AI on 2026-07-27. All three engines shipped K3-capable containers dated the same day, and the MAD recipes benchmarked here landed two days later.\n\n| Engine | Container image (tag as run) | Day-0 evidence | \n|---|---|---|\n| vLLM | `vllm/vllm-openai-rocm:kimi-k3` | [vLLM day-0 blog, 2026-07-27](https://vllm.ai/blog/2026-07-27-k3) and[MI355X recipe](https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi355x) | \n| SGLang | `lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727` | [day-0 tracking issue #32548](https://github.com/sgl-project/sglang/issues/32548) (support PR #32541) and[SGLang cookbook](https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3) | \n| ATOM | `rocm/atom-dev:rocm7.2.4_ubuntu24.04_py3.12_pytorch2.10.0_20260727_kimi_k3` | [Kimi-K3 on AMD Instinct GPUs](https://www.amd.com/en/developer/resources/technical-articles/2026/kimi-k3-on-amd-instinct-gpus.html) | \n| MAD | — | Kimi-K3 support merged in [MAD PR #186](https://github.com/ROCm/MAD/pull/186) , 2026-07-29 | \n\nTable 1: Day-0 enablement evidence per engine. The `20260727` tag suffix on the SGLang\nand ATOM images is the build date of the K3-enabled container.\n\n### System Configuration[#](#system-configuration)\n\nEvery measurement in this post was collected on the same node, described below.\n\n| Component | Value | \n|---|---|\n| System | Supermicro AS-8126GS-TNMR (H14DSG-OD baseboard) | \n| GPUs | 8× AMD Instinct™ MI350X (gfx950), TP8 | \n| CPU | 2× AMD EPYC™ 9575F, 64 cores each | \n| Host memory | 3 TiB (24× 128 GiB DDR5-6400 RDIMM) | \n| Storage | 2× Micron 7450 3.84 TB NVMe (PCIe Gen 5) | \n| Host networking | 2× Broadcom BCM57508 (up to 200 GbE), 2× Intel X710, 2× Intel X550 | \n| GPU interconnect | Single node, no multi-node fabric used | \n| OS | Ubuntu 24.04 | \n| AMD ROCm™ version | 7.2.3 (`rocm-core 7.2.3.70203-90` ) | \n| amdgpu driver | 6.16.13 (`amdgpu-dkms 1:6.16.13.30300100-2303411.24.04` ) | \n| VBIOS / firmware | `113-M350-01-1K1-030A` (identical across all 8 GPUs) | \n| GPU details | gfx950, device ID `0x75a0` , 256 CUs, HSA runtime 1.18 | \n\nTable 2: Host and accelerator configuration for every measurement in this post. The ROCm, driver, and VBIOS values were captured by madengine’s environment collection during the benchmark runs themselves, not reconstructed afterward.\n\n### Pinned Versions[#](#pinned-versions)\n\nReproducing these numbers exactly requires pinning the software, not tracking `main`:\n\n| Component | Pin | \n|---|---|\n| MAD | commit `a20c885` (PR #186) | \n| madengine | tag `v2.1.2` | \n| Kimi-K3 model revision | `9f62e4e9fffbd0a83ddd60e1c209d828994b3569` | \n| vLLM image digest | `sha256:5aa7e626ff73672f5ca7aae46754570488c23d33ca1ac90756a1d2d1a3fe099b` | \n| SGLang image digest | `sha256:3c01f73fe23aebf4a8853de0899a70b75c2af6c0409d2331353847aac4d3f906` | \n| ATOM image digest | `sha256:04ce312d4124e3c7f8a62a321bbd2d3f07328855f362f8e6374bdc5f51afc233` | \n\nTable 3: Version pins. The image digests are the ones the builds actually resolved and\nbaked in, read back from the build logs, so they identify what ran rather than what the\ntag points at today. That distinction matters here: Docker Hub tags are mutable, and for\nboth the vLLM and SGLang tags a `docker manifest inspect` run during the same build\nsession already returned a different digest than the build had pulled: `a8798d4a…` for\nvLLM and `c75ce7a3…` for SGLang. Pull by digest, not by tag, to reproduce these runs.\nThe ATOM Dockerfile already pins by digest in-repo. The vLLM and SGLang Dockerfiles\nreference their base images by tag.\n\n## The Day-0 Benchmarking Problem[#](#the-day-0-benchmarking-problem)\n\nKimi-K3 is not a bigger Kimi-K2. As [vLLM’s day-0 announcement](https://vllm.ai/blog/2026-07-27-k3)\nnotes, it changes the serving problem along many axes at once: hybrid Kimi Delta\nAttention (KDA) plus full attention, Attention Residuals, 896 routed experts with 16\nactive per token, MXFP4 weights with the SiTU activation, and native vision. Each axis\nlands somewhere different in each engine’s stack.\n\nNow multiply that by three frameworks, each with its own conventions:\n\n| Concern | vLLM | SGLang | ATOM | \n|---|---|---|---|\n| Container image | `vllm/vllm-openai-rocm:kimi-k3` | `lmsysorg/sglang-rocm:...-k3-20260727` | `rocm/atom-dev:...20260727_kimi_k3` | \n| Server entrypoint | `vllm serve` | `sglang serve` | `python -m atom.entrypoints.openai_server` | \n| MoE selector env | `AITER_SITUV2_A8W4=1` | `AITER_SITUV2_A8W4=1` +`SGLANG_AITER_K3_OPT=1` | `AITER_FLYDSL_FORCE=1` +`ATOM_USE_TRITON_MOE=0` | \n| Attention flag | (engine default) | `--attention-backend triton` | `ATOM_USE_UNIFIED_ATTN=1` | \n| Reasoning parser | `--reasoning-parser kimi_k3` | `--reasoning-parser kimi_k3` | (not set) | \n| Benchmark client | `vllm bench serve` | `sglang.benchmark.serving` | `atom.benchmarks.benchmark_serving` | \n| Result JSON schema | `total_token_throughput` ,`median_ttft_ms` … | SGLang JSONL | ATOM `median_*_ms` | \n\nTable 4: The same Kimi-K3 workload expressed three different ways. Each engine has its\nown image, entrypoint, kernel-selection environment variables, client, and result\nformat. The verbatim commands behind this summary are in\n[Benchmark Methodology](#benchmark-methodology), and the full server-side settings are\nin Table 9.\n\nDoing this by hand means three sets of `docker run` invocations, three server launch\nsequences, three health-check loops, and three JSON parsers, plus a fourth,\nerror-prone step of hand-reconciling the outputs. Every one of those steps can\nintroduce a divergence that is invisible in the final number: a mismatched input\nlength, a different concurrency point, a missing environment flag that selects the\nslow MoE path.\n\nMAD automates those steps.\n\n## The MAD Automation Flow[#](#the-mad-automation-flow)\n\nMAD is built around a **declarative model registry**, `models.json`, and the\n**madengine** runner. A single entry fully describes how to build, run, and score a\nworkload, and one command executes the whole pipeline.\n\nFigure 1: The madengine execution pipeline. One registry entry drives all five stages.\nWhat changes between engines is which row of `models.json` you select.\n\nFor every model, madengine performs the same five steps: **Build → Start → Resolve →\nExecute → Report**, regardless of which engine sits underneath. That uniformity is the\nwhole point: the *operator experience* is the same across vLLM, SGLang, and ATOM, even\nthough the internals differ substantially.\n\n### The Registry Entry Is the Contract[#](#the-registry-entry-is-the-contract)\n\nHere are the fields that make Kimi-K3-on-vLLM a one-command benchmark. The real entry\nalso carries bookkeeping fields, `url`, `owner`, `training_precision`, and `timeout`,\nomitted here for readability:\n\n```\n{\n  \"name\": \"pyt_vllm_kimi-k3\",\n  \"dockerfile\": \"docker/pyt_vllm_kimi_k3\",\n  \"scripts\": \"scripts/vllm/run.sh\",\n  \"data\": \"huggingface\",\n  \"n_gpus\": \"-1\",\n  \"multiple_results\": \"perf_Kimi-K3.csv\",\n  \"tags\": [\"pyt\", \"vllm\", \"inference\"],\n  \"skip_gpu_arch\": \"gfx942\",\n  \"args\": \"--model_repo moonshotai/Kimi-K3 --config configs/default.yaml\"\n}\n```\n\nThree engines, three near-identical entries, differing only in `dockerfile`,\n`scripts`, and `config`. The SGLang entry even ships **two variants**, `nospec` and\n`dspark` for speculative decoding, from the same script by passing `--variant`, and the\nsame `perf_Kimi-K3.csv` collects them all.\n\n```\n{ \"name\": \"pyt_sglang_kimi-k3\",\n  \"scripts\": \"scripts/sglang/run_kimi_k3.sh\",\n  \"args\": \"--model_repo moonshotai/Kimi-K3 --config configs/kimi_k3.yaml --variant nospec\" }\n\n{ \"name\": \"pyt_sglang_kimi-k3_dspark\",\n  \"scripts\": \"scripts/sglang/run_kimi_k3.sh\",\n  \"args\": \"--model_repo moonshotai/Kimi-K3 --config configs/kimi_k3.yaml --variant dspark\" }\n```\n\n### What’s Actually Running Under Those Five Stages[#](#whats-actually-running-under-those-five-stages)\n\nFigure 1 is the operator’s view. Figure 2 shows the same pipeline internally:\n\nFigure 2: madengine’s internal call chain for a Kimi-K3 run. The same classes handle every model in the registry.\n\n`madengine run` resolves the registry entry, builds an image from the entry’s\n`dockerfile`, reads the host GPU architecture from `rocminfo`, which reports `gfx950`\non MI350X and MI355X and `gfx942` on MI300X and MI325X, and checks it against\n`skip_gpu_arch`. It then resolves `MAD_DATAHOME` for the `\"data\": \"huggingface\"` entry,\nlaunches the container, and runs the entry’s `scripts` inside it. The script writes\n`perf_Kimi-K3.csv`. madengine passes the `multiple_results` value in as\n`MAD_OUTPUT_CSV`, and folds the result into the run-level `perf.csv`. None of this is\nKimi-K3-specific: enabling K3 meant adding `models.json` rows, Dockerfiles, and run\nscripts, with no changes to madengine itself. See the\n[madengine repository](https://github.com/ROCm/madengine) for the implementation.\n\n## The Config Is the Recipe[#](#the-config-is-the-recipe)\n\nThe benchmark recipe lives in version-controlled YAML, not in a person’s terminal history. Every server flag, every environment toggle that selects a kernel path, and every sweep axis is declarative and auditable.\n\nHere is the Kimi-K3 block of the vLLM config, `scripts/vllm/configs/default.yaml`,\nlightly abridged. The comments are condensed and a trailing `bench_args` block that\ndisables the GSM8K accuracy run is omitted:\n\n```\n- benchmark: serving\n  model: moonshotai/Kimi-K3\n  tp: 8\n  inp: 8192\n  out: 1024\n  dtype: auto\n  max_concurrency: 1 4 8 16 32 64 128 256      # the shared K3 sweep\n  env:\n    VLLM_ROCM_USE_AITER: 1\n    SAFETENSORS_FAST_GPU: 1\n    AITER_SITUV2_A8W4: 1                        # selects the aiter a8w4 MoE path\n    AITER_BF16_FP8_MOE_BOUND: 0\n    VLLM_USE_BREAKABLE_CUDAGRAPH: 0\n  extra_args:\n    --moe-backend: auto\n    --load-format: auto\n    --gpu-memory-utilization: 0.95\n    --mm-encoder-tp-mode: data                  # MoonViT-V2 is 401M; TP is pure overhead\n    --max-num-seqs: 256\n    --max-num-batched-tokens: 4096\n    --reasoning-parser: kimi_k3                  # K3 emits reasoning tokens by default\n    --language-model-only: true                 # text-only bench frees VRAM for KV\n```\n\nNotice that the comments encode why each knob is set: `AITER_SITUV2_A8W4: 1` selects\nthe AITER a8w4 MoE path, and `--mm-encoder-tp-mode: data` is set because MoonViT-V2 has\nonly 401M params, making TP on it pure communication overhead. The recipe is\nself-documenting, and re-running it a month later on a different cluster repeats the\nsame configuration, because the run parameters live in the config rather than in the\nshell it was launched from.\n\n## Harness Guardrails[#](#harness-guardrails)\n\nAutomation that produces wrong numbers quickly is worse than no automation. The MAD flow includes several guardrails so that a run that reports success is a run that measured what it claims to measure.\n\n### 1. Server-Health Gating Before Measurement[#](#server-health-gating-before-measurement)\n\nEvery serving runner launches the server as a subprocess and **polls it to readiness**\nbefore sending a benchmark request, so that the load time of a 1.56 TB checkpoint is not\ncounted as request latency:\n\n```\n# poll until the server is healthy, then start the benchmark client\nuntil curl -s http://localhost:8000/v1/models; do sleep 30; done\n```\n\nvLLM’s runner allows 30 minutes for that poll. Both SGLang and ATOM raise it to 5400\nseconds, appropriate for a multi-terabyte checkpoint. SGLang polls `/health` rather\nthan `/v1/models`, and ATOM’s `_wait_for_server()` also watches the server process\nitself, returning as soon as that process exits rather than waiting for the full\ntimeout to elapse.\n\n### 2. Common Core Output Schema[#](#common-core-output-schema)\n\nEvery engine, no matter its native JSON format, is parsed into a **common CSV core**,\nso downstream dashboards and regression checks do not need to special-case the engine:\n\n```\nmodel, benchmark, tp, inp, out, num_prompts,\nmax_concurrency, cmd, performance, metric, unit\n```\n\nEach runner adds a few engine-native columns on top of that shared core. vLLM adds\n`dtype` and `bs`. SGLang adds `variant`, for the `nospec` and `dspark` split, and\n`dtype`. ATOM adds `kv_cache_dtype`, `hf_pipeline_tag`, and `bs`. Each engine’s run\nproduces its own `perf_Kimi-K3.csv`, and `update_perf_csv` merges each into the\nrun-level `perf.csv`, carrying over any columns the base file doesn’t already have, so\nengine-specific columns are preserved even though the three engines don’t emit\nbyte-identical headers.\n\nThe runner records not just throughput but a set of common latency metrics:\n`median_ttft`, `median_tpot`, `median_itl`, `median_e2el`, plus the `cmd` that\nproduced the row, so each row in the CSV records the invocation that generated it.\nThese are medians. The CSV does not carry the full latency distribution, though each\nengine’s raw result JSON retains the percentiles requested via `--percentile-metrics`.\n\n| Metric | Meaning | Unit | \n|---|---|---|\n| `throughput_tot` | Total token throughput | tok/sec | \n| `throughput_gen` | Output (generation) throughput | tok/sec | \n| `median_ttft` | Time to first token | ms | \n| `median_tpot` | Time per output token | ms | \n| `median_itl` | Inter-token latency | ms | \n| `median_e2el` | End-to-end latency | ms | \n\nTable 6: The common core metric schema emitted for every engine and every concurrency point. Shared columns make cross-engine and cross-run comparison mechanical. Each engine adds its own extra columns on top.\n\n### 3. Reproducible Weights, Cached Once[#](#reproducible-weights-cached-once)\n\nThe `data: \"huggingface\"` field wires in weight resolution. By default, weights come\nfrom the Hub, using `hf-transfer` for speed and `MAD_SECRETS_HFTOKEN` for gated repos,\nbut `MAD_DATAHOME` transparently redirects to a pre-downloaded local copy, so the\nsame 1.56 TB checkpoint is fetched once and reused across every engine and every rerun:\n\n```\nmadengine run --tags pyt_vllm_kimi-k3 --keep-model-dir --live-output \\\n  --additional-context '{\"docker_mounts\": {\"/model_weights\": \"/path/to/Kimi-K3\"},\n                         \"docker_env_vars\": {\"MAD_DATAHOME\": \"/model_weights\"}}'\n```\n\n`--keep-model-dir` preserves that cache between runs. `--live-output` streams the\nunbuffered logs so a long sweep is observable in real time rather than a black box.\n\n### 4. Hardware-Aware Gating[#](#hardware-aware-gating)\n\nToday’s registry entries mark Kimi-K3 `skip_gpu_arch: gfx942`, because the day-0\nrecipes on all three engines assume the model’s native MXFP4 weights sit on the gfx950\ngeneration (MI350X and MI355X) and run a dense TP8 layout:\n\n```\n\"skip_gpu_arch\": \"gfx942\"\n```\n\nmadengine reads the host’s `MAD_SYSTEM_GPU_ARCHITECTURE`, the ROCm architecture string\n`rocminfo` reports (`gfx942` on MI300X and MI325X), and skips the workload rather than\nsilently producing a result under the wrong assumptions. That gate records the\nvalidated scope of this recipe: the configurations benchmarked here were written and\nvalidated for gfx950, and `skip_gpu_arch` keeps them from running outside it. madengine’s\n`arch_overrides` block is the general mechanism the shipped configs use to vary settings\nby GPU architecture.\n\n## Bring Your Own Sweep: Custom ISL and OSL Settings[#](#bring-your-own-sweep-custom-isl-and-osl-settings)\n\nThe shared ISL 8192, OSL 1024 sweep exists to align the three engines with each other\nand with the framework authors’ published figures. It is almost certainly not your\nworkload. A summarization service runs long input and short output. A code assistant\nruns the reverse. An agentic loop runs neither. Kimi-K3’s headline spec is a 1M-token\ncontext, which 8192 barely touches. This post contains no long-context data: `inp` has\nnot been pointed at 32k, 128k, or beyond on any of the three engines. That is the\nnatural next data point, and the harness below is what you would use to generate it.\n\n### 1. Copy the Config, Change the Shape[#](#copy-the-config-change-the-shape)\n\nIn your clone of MAD, copy the shipped K3 block into a new file next to it. The\n`configs/` directory alongside the runner is the path the container will look in:\n\n```\ncp scripts/vllm/configs/default.yaml scripts/vllm/configs/custom.yaml\n```\n\nThen trim it to the single block you care about and change the workload axes:\n\n```\n# scripts/vllm/configs/custom.yaml — a 2k/2k sweep instead of the shared 8k/1k\n- benchmark: serving\n  model: moonshotai/Kimi-K3\n  tp: 8\n  inp: 2048          # your input sequence length\n  out: 2048          # your output sequence length\n  dtype: auto\n  max_concurrency: 1 8 32 64        # your concurrency points\n  env:\n    VLLM_ROCM_USE_AITER: 1\n    SAFETENSORS_FAST_GPU: 1\n    AITER_SITUV2_A8W4: 1            # keep this — dropping it silently\n    AITER_BF16_FP8_MOE_BOUND: 0     # falls back to the slower a16w4 MoE path\n    VLLM_USE_BREAKABLE_CUDAGRAPH: 0\n  extra_args:\n    --moe-backend: auto\n    --gpu-memory-utilization: 0.95\n    --max-num-seqs: 256\n    --max-num-batched-tokens: 4096\n    --reasoning-parser: kimi_k3\n    --language-model-only: true\n```\n\nThe `env` and `extra_args` blocks are the tuned part of the recipe. Carry them over\nverbatim unless you are deliberately measuring one of those knobs. `AITER_SITUV2_A8W4`\nin particular selects the fast MoE kernel path. A “custom config” that quietly omits it\nproduces numbers that look like a regression, but the cause is really a misconfiguration.\n\n### 2. Point a Run at It[#](#point-a-run-at-it)\n\n`--additional-context` overrides the registry entry for a single invocation:\n\n```\nmadengine run --tags pyt_vllm_kimi-k3 --keep-model-dir --live-output \\\n  --additional-context '{\"model_args\": \"--model_repo moonshotai/Kimi-K3 --config configs/custom.yaml\",\n                         \"docker_mounts\": {\"/model_weights\": \"/shareddata/Kimi-K3\"},\n                         \"docker_env_vars\": {\"MAD_DATAHOME\": \"/model_weights\"}}'\n```\n\nTwo things are easy to get wrong here:\n\n- **`model_args` replaces the registry’s `args` string: it does not merge with it.** Whatever you pass is the complete argument list handed to the run script, so`--model_repo moonshotai/Kimi-K3` has to be restated alongside your`--config` .\nPassing only`--config configs/custom.yaml` leaves the model repo empty and the run\nscript exits on a missing argument.\n- **The whole thing is one JSON object.** All three keys,`model_args` ,`docker_mounts` ,\nand`docker_env_vars` , live inside a single pair of braces in a single pair of quotes.\n\nThere is also a shorthand. All three run scripts accept `CONFIG` as an environment\nvariable, so you can select a config without restating the model repo at all:\n\n```\nmadengine run --tags pyt_vllm_kimi-k3 --keep-model-dir --live-output \\\n  --additional-context '{\"docker_env_vars\": {\"CONFIG\": \"configs/custom.yaml\",\n                                             \"MAD_DATAHOME\": \"/model_weights\"},\n                         \"docker_mounts\": {\"/model_weights\": \"/shareddata/Kimi-K3\"}}'\n```\n\n### 3. Where the File Has to Live[#](#where-the-file-has-to-live)\n\nThe config path is resolved inside the container, relative to the scripts directory\nthat madengine copies in, so `configs/custom.yaml` means\n`scripts/vllm/configs/custom.yaml` in your checkout. A YAML sitting in `/tmp` on the\nhost will not be found. If you would rather not put the file in the repo, mount it and\npass an absolute container path instead:\n\n```\n--additional-context '{\"docker_mounts\": {\"/cfg\": \"/home/me/sweeps\"},\n                       \"docker_env_vars\": {\"CONFIG\": \"/cfg/custom.yaml\"}}'\n```\n\nThe same three flags work for the other two engines. The difference is which\nconfig the entry starts from: `scripts/sglang/configs/kimi_k3.yaml` for SGLang, which\nalso takes `--variant nospec|dspark`, and `scripts/atom/configs/default.yaml` for ATOM.\n\n### What You Give Up[#](#what-you-give-up)\n\nA custom sweep is no longer comparable to Figure 3, Table 10, or the framework tracking\nissue: those numbers apply at ISL 8192, OSL 1024. That is a fair trade when the\nquestion is “how does K3 serve my traffic on this node,” and the wrong tool when the\nquestion is “is this engine faster than that one.” Keep the shared sweep for the second\nquestion. Because the core schema is shared, both sets of numbers land in the same\n`perf_Kimi-K3.csv` shape, so you can carry both.\n\n| Knob | What it changes | Watch out for | \n|---|---|---|\n| `inp` /`out` | Input / output sequence length | Long `inp` raises KV pressure and may need a lower`max_concurrency` | \n| `max_concurrency` | Sweep points (space-separated) | Each value is a full server-side run, so cost scales linearly | \n| `tp` | Tensor parallel degree | TP8 is the configuration measured here, and lower TP degrees were not evaluated | \n| `extra_args` | vLLM server flags | Passed through verbatim to `vllm serve` | \n| `env` | Kernel-path selection | Dropping `AITER_SITUV2_A8W4` costs real throughput | \n\nTable 7: The knobs most worth editing in a custom config.\n\n## Benchmark Methodology[#](#benchmark-methodology)\n\nThis section states exactly how the numbers in Figure 3 and Table 10 were produced: what\nthe workload is, what commands ran, how each metric is defined, and how a row in the CSV\nbecomes a point on the chart. Everything below is emitted by the runner scripts from the\nconfigs shown earlier, so re-running the commands in [Get Started](#get-started)\nregenerates it.\n\n### Workload Definition[#](#workload-definition)\n\nAll three engines are driven with **synthetic random prompts**, not a natural-text\ndataset, so the workload is fully specified by four numbers and needs no external data\ndownload:\n\n| Property | Value | Notes | \n|---|---|---|\n| Benchmark mode | Online serving (OpenAI-compatible HTTP endpoint) | Not offline batch throughput | \n| Dataset | `--dataset-name random` | Synthetic token sequences drawn from the model’s own tokenizer vocabulary | \n| Input sequence length (ISL) | 8192 tokens | vLLM and SGLang pin it exactly; ATOM samples over `[0.8 × 8192, 8193]` | \n| Output sequence length (OSL) | 1024 tokens | vLLM and ATOM force it with `--ignore-eos` ; SGLang can stop early on EOS | \n| Requests per point | `10 × max_concurrency` | 10 at concurrency 1, up to 1280 at concurrency 128 | \n| Concurrency points | 1, 4, 8, 16, 32, 64, 128 | One full run per point | \n| Request arrival | `--request-rate inf` | All requests are released at once; the client’s `--max-concurrency` semaphore is the only limiter, so the server is saturated for the whole measurement | \n| Sampling | vLLM `--temperature 0` ; SGLang and ATOM use their client defaults | Output *length* is fixed by OSL, so sampling affects content, not the token count measured | \n| Tensor parallelism | TP8 across 8× MI350X | Single node, single server process | \n| Prefix caching | Disabled on all three engines | Identical prompts must not be served from cache | \n\nTable 8: Workload definition for every data point in this post. `num_prompts` is derived\nas `10 × max_concurrency` by all three runners, which keeps each point roughly 10 batches\ndeep rather than a fixed prompt count that would be trivially short at high concurrency.\n\nTwo properties of the run loop matter for reproducibility. First, **each concurrency\npoint gets a fresh server**: the runner launches the server, polls it to readiness, runs\nthe client once, then sends an interrupt signal to the server and its child processes\nbefore moving to the next point. No KV cache, no compiled-graph state, and no scheduler state carry across\npoints. Second, **there is no warm-up phase on the serving path**, so each point includes\nwhatever first-request compilation or autotuning the engine performs. The offline\n`vllm bench latency` path in the same runner does use `--num-iters-warmup 3`, but that\npath is not used here.\n\n### Exact Commands, Per Engine[#](#exact-commands-per-engine)\n\nmadengine emits these from the configs shown earlier. They are reproduced here with the\nconcurrency-32 point substituted in, so `--max-concurrency 32` and `--num-prompts 320`\nare the only values that change across the sweep. Each engine’s pair of commands is also\nrecorded verbatim in the `cmd` column of `perf_Kimi-K3.csv`, joined by a semicolon.\n\n#### vLLM[#](#vllm)\n\n```\n# server\nVLLM_ROCM_USE_AITER=1 SAFETENSORS_FAST_GPU=1 AITER_SITUV2_A8W4=1 \\\nAITER_BF16_FP8_MOE_BOUND=0 VLLM_USE_BREAKABLE_CUDAGRAPH=0 \\\nvllm serve moonshotai/Kimi-K3 --dtype auto -tp 8 \\\n  --no-enable-prefix-caching --trust-remote-code --disable-uvicorn-access-log \\\n  --moe-backend auto --load-format auto --gpu-memory-utilization 0.95 \\\n  --mm-encoder-tp-mode data --max-num-seqs 256 --max-num-batched-tokens 4096 \\\n  --reasoning-parser kimi_k3 --language-model-only\n\n# client\nvllm bench serve --model moonshotai/Kimi-K3 \\\n  --percentile-metrics ttft,tpot,itl,e2el --dataset-name random \\\n  --ignore-eos --temperature 0 --trust-remote-code \\\n  --max-concurrency 32 --num-prompts 320 \\\n  --random-input-len 8192 --random-output-len 1024 \\\n  --save-result --result-filename Kimi-K3_serving_8_8192_1024_320_32.json\n```\n\n#### SGLang[#](#sglang)\n\nShown here is the `nospec` variant. The `dspark` variant is the same server plus\n`--speculative-algorithm DSPARK` and\n`--speculative-draft-model-path RadixArk/Kimi-K3-DSpark`:\n\n```\n# server\nSGLANG_USE_AITER=1 SGLANG_AITER_K3_OPT=1 AITER_FLYDSL_FORCE=1 AITER_SITUV2_A8W4=1 \\\nsglang serve --model-path moonshotai/Kimi-K3 --dtype bfloat16 --tp-size 8 \\\n  --trust-remote-code --host 127.0.0.1 --port 30000 \\\n  --attention-backend triton --mem-fraction-static 0.85 \\\n  --cuda-graph-max-bs-decode 256 --disable-radix-cache \\\n  --reasoning-parser kimi_k3 --tool-call-parser kimi_k3\n\n# client\npython3 -m sglang.benchmark.serving --backend sglang \\\n  --host 127.0.0.1 --port 30000 --model moonshotai/Kimi-K3 \\\n  --dataset-name random --random-input-len 8192 --random-output-len 1024 \\\n  --random-range-ratio 1.0 --max-concurrency 32 --num-prompts 320 \\\n  --output-file Kimi-K3_nospec_serving_8_8192_1024_320_32.jsonl\n```\n\n#### ATOM[#](#atom)\n\n```\n# server\nATOM_LOADER_USE_THREADPOOL=1 ATOM_LOADER_THREADPOOL_WORKERS=16 ATOM_SYNC_AFTER_LOAD=1 \\\nATOM_DIST_TIMEOUT_SECONDS=3600 ATOM_USE_TRITON_GEMM=1 AITER_USE_GROUPED_GEMM=0 \\\nATOM_USE_TRITON_MOE=0 AITER_FLYDSL_FORCE=1 AITER_FORCE_GFX1250=0 \\\nATOM_USE_UNIFIED_ATTN=1 ATOM_FORCE_ATTN_TRITON=1 \\\npython -m atom.entrypoints.openai_server --model moonshotai/Kimi-K3 -tp 8 \\\n  --kv_cache_dtype fp8 --trust-remote-code --max-model-len 16384 \\\n  --max-num-seqs 256 --max-num-batched-tokens 10240 \\\n  --gpu-memory-utilization 0.93 --block-size 128 --no-enable_prefix_caching\n\n# client\npython -m atom.benchmarks.benchmark_serving --model moonshotai/Kimi-K3 \\\n  --backend vllm --base-url http://localhost:8000 \\\n  --percentile-metrics ttft,tpot,itl,e2el --dataset-name random \\\n  --ignore-eos --request-rate inf --random-range-ratio 0.8 \\\n  --max-concurrency 32 --num-prompts 320 \\\n  --random-input-len 8192 --random-output-len 1024 \\\n  --save-result --result-dir ./ \\\n  --result-filename Kimi-K3_serving_8_8192_1024_320_32.json --trust-remote-code\n```\n\n### Server-Side Settings Side by Side[#](#server-side-settings-side-by-side)\n\nThe workload axes in Table 8 are aligned across engines. The server-side memory and\nscheduling knobs are each engine’s day-0 recipe values and are **not** normalized, which\nis part of why these results are an out-of-box snapshot rather than an engine comparison:\n\n| Server setting | vLLM | SGLang | ATOM | \n|---|---|---|---|\n| Weight/activation dtype | `auto` (native MXFP4) | `bfloat16` | not set (native MXFP4) | \n| KV cache dtype | engine default (bf16) | engine default (bf16) | `fp8` | \n| GPU memory budget | `--gpu-memory-utilization 0.95` | `--mem-fraction-static 0.85` | `--gpu-memory-utilization 0.93` | \n| Max running sequences | `--max-num-seqs 256` | `--cuda-graph-max-bs-decode 256` | `--max-num-seqs 256` | \n| Max batched tokens | 4096 | engine default | 10240 | \n| Max model length | engine default | engine default | 16384 | \n| KV block size | engine default | engine default | 128 | \n| Attention backend | engine default | `triton` | `ATOM_USE_UNIFIED_ATTN=1` ,`ATOM_FORCE_ATTN_TRITON=1` | \n| MoE kernel path | `AITER_SITUV2_A8W4=1` (AITER a8w4) | `AITER_SITUV2_A8W4=1` +`SGLANG_AITER_K3_OPT=1` | `ATOM_USE_TRITON_MOE=0` +`AITER_FLYDSL_FORCE=1` | \n| Multimodal tower | `--language-model-only` (MoonViT-V2 not loaded) | not loaded by default | not loaded by default | \n| Speculative decoding | off | off (`nospec` ); enabled by the`_dspark` entry | off | \n\nTable 9: Server-side configuration as run. The differing memory budgets, KV-cache dtype, and batched-token limits are day-0 recipe defaults per engine. They are material to throughput at high concurrency, where KV capacity sets how many of the requested concurrent sequences actually run rather than queue.\n\n### Metric Definitions and Data Reduction[#](#metric-definitions-and-data-reduction)\n\nThe reported number is each client’s own **total token throughput**, not a quantity\nrecomputed by MAD:\n\n- **Total token throughput** (`throughput_tot` , tok/s) is`(total prompt tokens + total generated tokens) / benchmark duration` , where the\nduration spans from the first request dispatch to the last request completion. vLLM\nreports it as`total_token_throughput` , SGLang as`total_throughput` ; the runners copy\nthe field through unchanged.\n- **Output throughput** (`throughput_gen` , tok/s) counts generated tokens only, over the\nsame window.\n- **`median_ttft`, `median_tpot`, `median_itl`, `median_e2el`** are the medians of the\nper-request distributions the clients compute under`--percentile-metrics ttft,tpot,itl,e2el` .\nOnly the median reaches the CSV; the full percentile set stays in each engine’s raw\nresult JSON next to it.\n\nData reduction for Figure 3 and Table 10 is a filter, not an aggregation: for each engine\nand each concurrency point, take the single row of `perf_Kimi-K3.csv` whose `metric`\ncolumn is `throughput_tot`, and plot `performance` against `max_concurrency`. Because\neach point is one run, there is nothing to average and no error bar to draw. The GSM8K\naccuracy stage is switched off for vLLM in the config (`--run_accuracy: False`), because\nK3’s always-on reasoning tokens exhaust `lm_eval`’s hardcoded 2048-token generation\nbudget over `/v1/completions` and produce a meaningless score, so no accuracy number is\nclaimed anywhere in this post.\n\n### Reproduction Checklist[#](#reproduction-checklist)\n\nTo land on comparable numbers, the elements that must match are, in rough order of how much they move the result:\n\n1. **Hardware and software stack:** 8× MI350X (gfx950), ROCm 7.2.3, amdgpu 6.16.13\n(Table 2).\n2. **Images by digest, not tag** (Table 3): both the vLLM and SGLang tags moved within a\nsingle build session.\n3. **Model revision**`9f62e4e9…` , fetched once and shared via`MAD_DATAHOME` .\n4. **MAD `a20c885` and madengine `v2.1.2`** , so the runners emit the commands above.\n5. **The stock configs** , unedited, especially the`env` blocks: dropping`AITER_SITUV2_A8W4` silently selects a slower MoE path and looks like a regression.\n6. **One server per concurrency point** , which the runner does for you.\n\n## Results: Out-of-Box, All Three Engines[#](#results-out-of-box-all-three-engines)\n\nReproducing the day-0, three-engine benchmark is three commands:\n\n```\n# vLLM — 8k/1k serving sweep, TP8\nmadengine run --tags pyt_vllm_kimi-k3   --keep-model-dir --live-output\n\n# SGLang — same sweep; add the _dspark tag for speculative decoding\nmadengine run --tags pyt_sglang_kimi-k3 --keep-model-dir --live-output\n\n# ATOM — same sweep, fp8 KV cache\nmadengine run --tags pyt_atom_kimi-k3   --keep-model-dir --live-output\n```\n\nEach produces a `perf_Kimi-K3.csv` sharing the same core columns, so stacking the three\nCSVs lines the concurrency axis up row-for-row.\n\nBelow is an out-of-box `madengine run` on 8× MI350X, all three engines, no tuning beyond\nthe shared config in this post.\n\nFigure 3: **Out-of-box snapshot.** Total token throughput vs. max concurrency, 8192 in,\n1024 out, TP8, 8× MI350X, measured 2026-07-29. All three engines are from the same madengine\nsweep, each using its day-0 out-of-box configuration, with one run per concurrency\npoint and no repetitions. This is not a tuned comparison and not a leaderboard: it is a\npicture of how each stack’s day-0 out-of-box configuration behaved on launch day, and it\nis the thing most likely to have changed by the time you re-run the command.\n\nTwo properties of the shape matter more than the ordering. First, this is a functional result before it is a performance one: a 2.8T-parameter MoE with a new attention design completes the full sweep on a single 8× MI350X node, on three separate engines, on day 0. No accuracy evaluation was run as part of this sweep, so these results establish that the model serves and produces output at the requested lengths, not that its output quality was assessed. Second, at concurrency 32 all three land on essentially the same point: 4,676, 4,694, and 4,692 tok/s, a spread under 0.5%. The spread is considerably wider below that point: at concurrency 1 the three values differ by more than 40%, and it widens again above concurrency 32. That single-point convergence is an observation about this configuration. One throughput point does not isolate which component sets the ceiling there: scheduler, kernels, precision, benchmark client, or hardware.\n\nThe curves separate past concurrency 32, where the engines’ differing scheduling and batching defaults have more room to act. Because each engine runs its own out-of-box day-0 configuration, these high-concurrency values reflect those default settings rather than a tuned result for any engine, and the differences between the three benchmark clients noted in Table 5 remain unquantified. Each engine’s day-0 configuration is a snapshot of one point in its development, and all three continue to change.\n\n| Concurrency | vLLM (tok/s) | SGLang (tok/s) | ATOM (tok/s) | \n|---|---|---|---|\n| 1 | 287.28 | 422.75 | 346.12 | \n| 4 | 1,000.47 | 1,388.58 | 1,187.52 | \n| 8 | 1,744.78 | 2,263.67 | 2,056.96 | \n| 16 | 2,985.02 | 3,451.65 | 3,297.54 | \n| 32 | 4,675.92 | 4,693.86 | 4,691.54 | \n| 64 | 6,567.25 | 5,994.97 | 5,024.71 | \n| 128 | 8,228.15 | 6,293.32 | 5,136.26 | \n\nTable 10: **Out-of-box snapshot.** Raw total-token-throughput values behind Figure 3,\nstraight out of each engine’s `perf_Kimi-K3.csv`. Measured 2026-07-29 on 8× MI350X, one\nrun per concurrency point, no repetitions, so these values carry no run-to-run variance\nestimate. The shipped configs list concurrency 256, but the runs reported here were taken\nthrough 128 only. 256 has not been measured on any of the three engines.\n\n## Limitations[#](#limitations)\n\nThe measurements above are a day-0 snapshot, and the following bound what they support:\n\n- **MI350X only.** The recipes target gfx950 (MI350X and MI355X), but every number here\ncomes from an 8× MI350X node. MI355X has not been measured.\n- **Throughput only, no accuracy evaluation.** The sweep measures serving throughput and\nlatency; the GSM8K accuracy run is disabled in the configs used here, so output quality\nwas not assessed on any of the three engines.\n- **Single run per point.** Each concurrency point is one measurement over`10 × concurrency` prompts, with no repetitions and no warm-up iterations on the\nserving path. There is no run-to-run variance estimate, so small cross-engine\ndifferences should not be over-read.\n- **Concurrency 256 unmeasured.** The shipped configs list it; no engine has run it, so\nthe config’s sweep list is wider than the measured range reported here.\n- **Bench-client defaults not normalized.** The prompt-length sampling and EOS\ndifferences in Table 5 are unquantified. No A/B run has isolated their effect.\n- **No long-context data.** Kimi-K3 supports a 1M-token context; the sweep uses 8192.\nLong-context behavior on all three engines is unmeasured.\n- **Mutable image tags.** The digests in Table 3 identify exactly what ran, but the vLLM\nand SGLang Dockerfiles reference their base images*by tag* , and both tags were\nobserved serving a different digest within the same build session. Anyone rebuilding\nfrom the tag today may not get the image benchmarked here. The upstream framework\ncommits inside those containers are also not recorded.\n\n## The Same Pattern Across the MAD Catalog[#](#the-same-pattern-across-the-mad-catalog)\n\nThe same registry-plus-runner pattern used for Kimi-K3 already spans the AMD MAD\ncatalog: vLLM, SGLang, ATOM, Primus and Megatron training, JAX MaxText, xDiT diffusion,\nand disaggregated prefill and decode (P/D) serving, all driven by the same\n`madengine run --tags …` interface and the same declarative configs.\n\nThat uniformity makes day-0 support repeatable rather than a one-off:\n\n- **For model launches:** enabling a new model on a new engine is a registry entry, a\nDockerfile, and a YAML, reviewable in a PR, not lost in a shell session.\n- **For CI and regression:** the shared core schema and shared sweeps mean a nightly job\ncan diff today’s`perf_Kimi-K3.csv` against a reference and flag drift automatically.\n- **For the community:** anyone with an 8× MI350X node can reproduce this benchmark from\nthe configs in the repo, subject to the version pins in Table 3.\n\nAs Moonshot AI and the framework teams extend agentic serving, with longer horizons, deeper tool use, and larger context, the same harness is what makes each step measurable on AMD Instinct™ hardware.\n\n## Get Started[#](#get-started)\n\nTo reproduce the runs in this post, pin both repositories to the versions in Table 3\nrather than tracking `main`:\n\n```\npip install git+https://github.com/ROCm/madengine.git@v2.1.2\ngit clone https://github.com/ROCm/MAD.git && cd MAD\ngit checkout a20c885\n\n# pick your engine\nmadengine run --tags pyt_vllm_kimi-k3   --keep-model-dir --live-output\nmadengine run --tags pyt_sglang_kimi-k3 --keep-model-dir --live-output\nmadengine run --tags pyt_atom_kimi-k3   --keep-model-dir --live-output\n```\n\nTo pin the checkpoint to the exact revision benchmarked here, fetch it once by revision\nand point `MAD_DATAHOME` at the result:\n\n```\nhf download moonshotai/Kimi-K3 \\\n  --revision 9f62e4e9fffbd0a83ddd60e1c209d828994b3569 \\\n  --local-dir /shareddata/Kimi-K3\n```\n\n- **madengine:**[github.com/ROCm/madengine](https://github.com/ROCm/madengine)\n- **Model:**[moonshotai/Kimi-K3 on Hugging Face](https://huggingface.co/moonshotai/Kimi-K3)\n\n## Summary[#](#summary)\n\nIn this blog you explored how MAD’s declarative model registry and its madengine runner turn day-0 Kimi-K3 enablement on AMD Instinct™ MI350X into a single reproducible command per engine. You saw what a registry entry contains and how madengine expands it into the Build, Start, Resolve, Execute, and Report pipeline; how one shared sweep of 8192 input and 1024 output tokens at TP8 across seven concurrency points is expressed in versioned YAML rather than in shell history; how the harness gates on server health, caches weights, and guards against unvalidated GPU architectures; and how to retarget the whole sweep to your own ISL, OSL, and concurrency by copying one config. You also saw the out-of-box day-0 numbers the flow produced for vLLM, SGLang, and ATOM, together with an explicit account of what one run per point, with server settings that are not normalized across engines, does and does not establish.\n\nThe same registry-plus-runner pattern already covers training, diffusion, and\ndisaggregated prefill and decode serving across the MAD catalog, so the next step is to\nrun it on your own cluster: clone MAD at the pinned commit, launch the engine you care\nabout, and diff your `perf_Kimi-K3.csv` against the configuration published here. Our\nteam will keep extending this harness as agentic serving workloads grow longer horizons\nand larger context, and future posts will cover tuned configurations, multi-node\nscaling, and disaggregated serving on AMD Instinct™ GPUs.\n\n## Additional Resources[#](#additional-resources)\n\n- [Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) : Moonshot AI’s 2.8T-parameter Mixture-of-Experts LLM\n- [MAD](https://github.com/ROCm/MAD) : Model Automation and Dashboarding for AMD Instinct GPUs\n- [madengine](https://github.com/ROCm/madengine) : The MAD execution engine and CLI\n- [vLLM](https://github.com/vllm-project/vllm) : High-throughput serving engine for large language models\n- [SGLang](https://github.com/sgl-project/sglang) : Fast serving framework for large language models\n- [AITER](https://github.com/ROCm/aiter) : AI Tensor Engine for ROCm\n\n## Disclaimers[#](#disclaimers)\n\nTesting conducted by AMD on 2026-07-29. Hardware configuration: Supermicro\nAS-8126GS-TNMR with 8× AMD Instinct™ MI350X (gfx950), 2× AMD EPYC 9575F, 3 TiB DDR5\nsystem memory, Ubuntu 24.04, ROCm 7.2.3, amdgpu driver 6.16.13, TP8. Kimi-K3 checkpoint\n≈ 1.56 TB (revision `9f62e4e9`). The recipes target the\ngfx950 generation (MI350X and MI355X). MI355X was not measured. Workload: online serving\nof synthetic random prompts, 8192 input tokens and 1024 output tokens, `10 × concurrency`\nrequests per point, request rate `inf`, prefix caching disabled, concurrency swept over\n1/4/8/16/32/64/128, with a fresh server process per concurrency point and no warm-up.\nReported throughput is each benchmark client’s own total token throughput. Results\nreflect out-of-box engine configurations with one run per data point and no repetitions;\nserver-side settings are not normalized across engines. See Tables 2 and 3 for the full\nsystem and version configuration, Tables 8 and 9 for the workload and server\nconfiguration, and the verbatim commands in the Benchmark Methodology section.\n\nThird-party content is licensed to you directly by the third party that owns the content and is not licensed to you by AMD. ALL LINKED THIRD-PARTY CONTENT IS PROVIDED “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF SUCH THIRD-PARTY CONTENT IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THIRD-PARTY CONTENT.\n\nResults shown are from specific test configurations and may vary based on workload, model, and system configuration.\n\nThe information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED “AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, AMD Instinct, AMD EPYC, AMD ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. PyTorch, the PyTorch logo, vLLM, ubuntu, and any related marks are trademarks of The Linux Foundation. All other trademarks and product names referenced in this publication, including Kimi-K3, Moonshot AI, SGLang, and Hugging Face, are the property of their respective owners. © 2026 Advanced Micro Devices, Inc. All rights reserved.", "url": "https://wpnews.pro/news/benchmarking-kimi-k3-across-vllm-sglang-and-atom-on-mi350x", "canonical_source": "https://rocm.blogs.amd.com/artificial-intelligence/kimi-k3-mad/README.html", "published_at": "2026-09-22 00:00:00+00:00", "updated_at": "2026-09-22 17:26:26.963423+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-chips", "mlops", "ai-research"], "entities": ["Moonshot AI", "Kimi-K3", "AMD", "AMD Instinct MI350X", "vLLM", "SGLang", "ATOM", "MAD"], "alternates": {"html": "https://wpnews.pro/news/benchmarking-kimi-k3-across-vllm-sglang-and-atom-on-mi350x", "markdown": "https://wpnews.pro/news/benchmarking-kimi-k3-across-vllm-sglang-and-atom-on-mi350x.md", "text": "https://wpnews.pro/news/benchmarking-kimi-k3-across-vllm-sglang-and-atom-on-mi350x.txt", "jsonld": "https://wpnews.pro/news/benchmarking-kimi-k3-across-vllm-sglang-and-atom-on-mi350x.jsonld"}}