{"slug": "a-megakernel-per-model", "title": "A Megakernel per Model", "summary": "A coding-agent framework that writes a single megakernel per model checkpoint per GPU type produced inference engines 1.13–1.77× faster than the fastest general engine (vLLM or SGLang) with one sequence across all eight tested models from 8B to 594B parameters, and up to 2.05× when model storage may change without altering its numerical function. On a production trace with Inkling-2-Small on 4× B200 at 1 to 8 requests per second, 99–100% of requests stayed within service levels versus 82–85% for SGLang, and the longest stall was 0.08–0.15 s versus 2.3–4.4 s. The framework, run with Claude Opus 5.5 and Fable 5.2, gates timing behind a test suite the agent cannot change, requiring numerical agreement with a high-precision reference on unseen inputs during actual execution plus an independent audit; the engines, an OpenAI-API server, the test suite and reproduction tools are released in megakernel-zoo.", "body_md": "General inference engines such as vLLM<sup>[1](#ref-1)</sup> and SGLang<sup>[2](#ref-2)</sup> serve every model through one scheduler and one library of kernels. We let a coding agent write the engine for one checkpoint on one GPU instead, as a **megakernel**: one GPU program that runs the whole decode loop without returning to the host between tokens.vLLM<sup>[1](#ref-1)</sup> 和 SGLang<sup>[2](#ref-2)</sup> 这样的通用 inference engine 用同一个 scheduler 和同一套 kernel 库来服务所有模型。我们反过来，让一个 coding agent 为一块 GPU 上的一个 checkpoint 写专用的 engine，形式是一个 **megakernel**：一个 GPU 程序跑完整个 decode loop，token 之间不回到 host。\n\n**Steady decoding.** With the same data flow as the reference implementation (track A), the engines are 1.13–1.77× as fast as the fastest general engine with one sequence, on all eight models from 8 B to 594 B parameters. When the storage of the model may change but its function may not, numerically (track B), the factor reaches 2.05×. **Serving a production trace** on Inkling-2-Small and 4 × B200, at 1 to 8 requests per second, 99–100% of requests stay within the service levels against 82–85% for SGLang, and the longest stall is 0.08–0.15 s against 2.3–4.4 s.**Steady decoding。** 在数据流与参考实现相同的前提下（track A），单序列时 engine 的速度是最快通用 engine 的 1.13–1.77 倍，覆盖从 8 B 到 594 B 参数的全部八个模型。如果允许改变模型的存储方式、但不改变它在数值上的函数（track B），倍数可达 2.05×。**在生产 trace 上做 serving**：Inkling-2-Small 跑在 4 × B200 上，每秒 1 到 8 个请求时，99–100% 的请求满足 service level，SGLang 是 82–85%；最长卡顿是 0.08–0.15 s，SGLang 是 2.3–4.4 s。\n\n**Validation.** We built a framework in which agents design megakernels on their own. Its core is a test suite the agent cannot change: an engine is timed only after it agrees with a high-precision reference in its numbers, on inputs it has never seen and while it actually runs, and after an independent audit. We ran it with Claude Opus 5.5 and Fable 5.2. The engines, a server that puts them behind the OpenAI API, the test suite and the tools to run it on another model are in [megakernel-zoo](https://github.com/dangxingyu/megakernel-zoo).**验证。** 我们搭了一个让 agent 自主设计 megakernel 的框架，核心是一套 agent 改不了的 test suite：engine 必须在数值上与高精度 reference 一致——包括它从没见过的输入、包括它真正运行时的输出——并通过独立审计之后，才会被计时。我们用 Claude Opus 5.5 和 Fable 5.2 跑了这套框架。engine、把它们包装成 OpenAI API 的 server、test suite，以及在别的模型上复现的工具，都在 [megakernel-zoo](https://github.com/dangxingyu/megakernel-zoo) 里。\n\n## A megakernel per model每个模型一个 megakernel\n\nvLLM and SGLang are designed to run any model. Each decoding step is assembled from a library of general-purpose kernels, one per operation (attention, matrix multiplication, normalization), which run one after another from a CUDA graph, while the CPU schedules every step. We take the other extreme: **an agent writes a single kernel for one checkpoint on one GPU type**, and that kernel executes every operation of every step, and the loop over tokens, by itself.vLLM 和 SGLang 的设计目标是能跑任何模型。每个 decoding step 由一个通用 kernel 库拼成，每种运算一个 kernel（attention、矩阵乘、normalization），这些 kernel 从一个 CUDA graph 里依次执行，由 CPU 调度每一个 step。我们走另一个极端：**由 agent 为一种 GPU 上的一个 checkpoint 写一个 kernel**，这个 kernel 自己执行每个 step 的所有运算，以及对 token 的循环。\n\nWith the whole decoding step in one kernel, the agent can schedule operators freely: prefetch the next layer's weights while the current layer computes, fuse across operator boundaries without round-tripping intermediates through memory, and start idle thread blocks on the next phase early. Separate kernel launches make each of these hard.整个 decoding step 都在一个 kernel 里，agent 就能自由地调度算子：当前层计算时预取下一层的权重，跨算子边界融合而不必让中间结果经过显存往返，让空闲的 thread block 提前开始下一阶段。拆成多次 kernel launch 时，这些都很难做到。\n\nThe price is why megakernels are rare: they are hard to maintain and debug, they are tied to one model and one kind of GPU, and they need retuning when the batch or context changes, so until now they have been written by hand for a few models (Hazy Research's for Llama-1B<sup>[3](#ref-3)</sup>, Cohere's for North Mini Code<sup>[4](#ref-4)</sup>) or compiled for a set of supported architectures (MPK<sup>[5](#ref-5)</sup>). Coding agents now have the capability to do this work, so that cost is gone.代价正是 megakernel 少见的原因：它难维护、难调试，绑定在一个模型和一种 GPU 上，batch 或 context 一变就要重新调优，所以至今只有少数几个模型有手写版本（Hazy Research 为 Llama-1B 写的<sup>[3](#ref-3)</sup>、Cohere 为 North Mini Code 写的<sup>[4](#ref-4)</sup>），或者由编译器为一组支持的架构生成（MPK<sup>[5](#ref-5)</sup>）。现在 coding agent 有 capability 来完成这个事情，所以没有这个成本了。\n\nWe report two tracks.我们分两个 track 报告。\n\n- **Track A** keeps**the data flow of the reference implementation unchanged** : weights in the checkpoint's own formats, activations and KV cache in the formats the general engines use; only how the computation is organized on the GPU changes.**Track A** 要求**数据流与参考实现保持一致** ：权重保持 checkpoint 自己的格式，activation 和 KV cache 用通用 engine 的格式；改变的只是计算在 GPU 上如何组织。\n- **Track B** is looser: it only requires the function to be**unchanged in exact arithmetic** , and lets the agent change how the model is stored and what is computed at load time. Lossy quantization or pruning does not qualify, since it changes the function.**Track B** 更宽松，只要求函数**在精确算术下不变** ，允许 agent 改变模型的存储方式和 load 时预先计算的内容。有损的 quantization 或 pruning 不算，因为它们改变了函数。\n\nWe evaluate eight models from 8 B to 594 B parameters, dense and mixture-of-experts, on H100 and B200 GPUs, against vLLM and SGLang and, for North Mini Code, Cohere's megakernel. Versions and flags are on the [data page](full.html).我们评测了八个模型，从 8 B 到 594 B 参数，有 dense 也有 mixture-of-experts，跑在 H100 和 B200 GPU 上，对比 vLLM 和 SGLang；North Mini Code 还对比了 Cohere 的 megakernel。版本和参数见 [数据页](full.html)。\n\n## Steady decodingSteady decoding\n\nWe use the standard decode benchmark: a 64-token prompt, 1,024 generated tokens, greedy decoding, batch size 1 or 8, every system served over HTTP on the same GPU.我们用标准的 decode benchmark：64-token 的 prompt，生成 1,024 个 token，greedy decoding，batch size 为 1 或 8，所有系统都通过 HTTP、在同一块 GPU 上提供服务。\n\nAt batch size 1, track A is 1.13 to 1.77 times as fast as the fastest general engine on every model. At batch size 8 the speedups are smaller on six of the eight models, lowest on North Mini Code (1.02×) and gpt-oss-120b (1.08×). On North Mini Code the fastest general engine is Cohere's megakernel; against vLLM, track A is 1.99× and 1.37×. On the engines also timed with prose and code prompts and with sampling, the speedups stay within 0.05 of the greedy random-token ones ([data page](full.html)).batch size 为 1 时，track A 在每个模型上都是最快通用 engine 的 1.13 到 1.77 倍。batch size 为 8 时，八个模型中有六个的加速比更小，最低的是 North Mini Code（1.02×）和 gpt-oss-120b（1.08×）。North Mini Code 上最快的通用 engine 是 Cohere 的 megakernel；对比 vLLM，track A 是 1.99× 和 1.37×。另外用散文和代码 prompt、以及 sampling 测过的 engine，加速比与 greedy 随机 token 的结果相差不超过 0.05（[数据页](full.html)）。\n\nBatch-1 decoding is bound by HBM bandwidth. On Qwen3-8B and an H100, reading the 15.1 GB of BF16 weights once at peak bandwidth (3.35 TB/s) takes 4.52 ms; the track-A engine runs at 5.13 ms per token, 88% of that bound.batch size 为 1 的 decoding 受 HBM 带宽限制。在 Qwen3-8B 和 H100 上，以峰值带宽（3.35 TB/s）把 15.1 GB 的 BF16 权重读一遍要 4.52 ms；track-A engine 每个 token 5.13 ms，达到这个上界的 88%。\n\n### What track B usedTrack B 用了什么\n\nIn track B, the agent can speed up the engine by observing features of this checkpoint's weights. The clearest example is the weight format: BF16 spends 8 bits on the exponent, yet the weights in one row of a trained matrix span only a few binades, a redundancy that lossless formats such as DFloat11<sup>[6](#ref-6)</sup> also exploit. The Qwen3-8B agent measured this, encoded each weight losslessly in 12 bits and decoded it in registers ahead of the tensor cores, cutting weight traffic per token by a quarter. Two other techniques recur in Table 1: a certified low-precision sketch of the LM head that shortlists the next token, and per-token lookup tables for layer 0, whose input depends only on the token's embedding. Concurrent work<sup>[7](#ref-7)</sup> found the same format: SpecPilot, an agentic loop that also runs Claude Opus 5.5 in Claude Code, optimizes speculative decoding for each deployment and lists this lossless 12-bit format among its discoveries. Our Claude Opus 5.5 agent found it independently, inside a full decode megakernel. Later runs adapted the method to Qwen3.8-27B and Qwen3-32B, where every product is bit-identical to BF16 arithmetic, and a Claude Fable 5.2 run applied it to the LM head of Inkling-2-Small.在 track B 中，agent 可以通过观察这个 checkpoint 的 weight 的一些 feature 来 speedup。最明显的例子是权重格式：BF16 用 8 个 bit 表示 exponent，但训练好的矩阵里同一行的权重只跨几个 binade，DFloat11<sup>[6](#ref-6)</sup> 这类无损格式利用的也是这种冗余。Qwen3-8B 上的 agent 测出了这一点，把每个权重无损地编码成 12 bit，并在送往 tensor core 之前于寄存器内解码，每个 token 的权重流量减少四分之一。表 1 里还反复出现两种技术：LM head 的 certified 低精度 sketch，用来筛出下一个 token 的候选；以及第 0 层的 per-token 查找表，因为这一层的输入只取决于 token 的 embedding。同期工作<sup>[7](#ref-7)</sup>发现了同样的格式：SpecPilot 是一个同样在 Claude Code 里运行 Claude Opus 5.5 的 agentic loop，它为每个部署优化 speculative decoding，并把这种无损 12-bit 格式列为它的发现之一。我们的 Claude Opus 5.5 agent 也在完整的 decode megakernel 内部独立发现了它。后面的 run 把这个 method adapt 到了 Qwen3.8-27B 和 Qwen3-32B 上，结果上每个乘积都与 BF16 算术逐位相同；一个 Claude Fable 5.2 的 run 也把它用到了 Inkling-2-Small 的 LM head 上。\n\n#### Table 1. Track B on each model表 1. 各模型上的 track B\n\n| Model模型 | Effect in the runrun 中测得的效果 | Logits against track A与 track A 的 logits 对比 | \n|---|---|---|\n| flintflint | −2.9% (head sketch)−2.9%（head sketch） | bit-identical逐位相同 | \n| Qwen3.8-27BQwen3.8-27B | −9% (12-bit weights), then −5.6%−9%（12-bit 权重），之后 −5.6% | bit-identical逐位相同 | \n| Qwen3-32BQwen3-32B | −11% (10.12 → 9.00 ms), 12-bit weights−11%（10.12 → 9.00 ms），12-bit 权重 | bit-identical逐位相同 | \n| gpt-oss-120bgpt-oss-120b | −5% (head sketch)−5%（head sketch） | bit-identical逐位相同 | \n| Inkling-2-SmallInkling-2-Small | −0.4% | bit-identical逐位相同 | \n| Qwen3-8BQwen3-8B | −15% (5.34 → 4.52 ms), 12-bit weights−15%（5.34 → 4.52 ms），12-bit 权重 | not bit-identical: the residual stream is kept in FP32 (BF16 in track A), so it is more precise非逐位相同：residual stream 保持 FP32（track A 是 BF16），因此精度更高 | \n| North Mini CodeNorth Mini Code | −10% (12-bit weights), −4% (head sketch)−10%（12-bit 权重），−4%（head sketch） | not bit-identical: the rare weights outside the 12-bit range are added in a separate pass, same values in a different rounding order非逐位相同：少数超出 12-bit 范围的权重单独补算，数值相同，舍入顺序不同 | \n| gpt-oss-20bgpt-oss-20b | −14% (1.87 → 1.61 ms)−14%（1.87 → 1.61 ms） | not bit-identical: the router weights are pre-multiplied by the norm weights in FP32, so the normalized input is never rounded to BF16 and the router logits differ in the last bits非逐位相同：router 权重预先乘上 norm 权重并以 FP32 存储，归一化后的输入不再舍入到 BF16，router logits 在最后几位不同 | \n\n[data page](full.html).引入每处改动的那次 submission 在 run 自己的 evaluator 下前后的分数。head sketch 是 LM head 的低精度副本，带有可证明的误差上界：它只用来筛选候选 token，所有报告的 logit 都来自精确的 head。每个 engine 具体做了什么见\n\n[数据页](full.html)。\n\nWhere the checkpoint is already compressed (the 4-bit experts of Inkling-2-Small and gpt-oss), track B gains little.如果 checkpoint 本身已经压缩过（Inkling-2-Small 和 gpt-oss 的 4-bit expert），track B 的收益很小。\n\n## Serving live traffic服务真实流量\n\nThe engines above decode a batch whose sequences start together; a server must admit new requests while others are decoding. For Qwen3-8B and Inkling-2-Small, a second run moved scheduling onto the GPU: continuous batching, chunked prefill inside the kernel, a paged KV cache and prefix caching.上面的 engine 解码的是同时开始的一个 batch；而 server 必须在其他请求 decode 的同时接收新请求。对 Qwen3-8B 和 Inkling-2-Small，第二个 run 把调度搬到了 GPU 上：continuous batching、kernel 内的 chunked prefill、paged KV cache 和 prefix caching。\n\nWe replay the Mooncake tool-and-agent trace<sup>[8](#ref-8)</sup> (mean prompt length about 9,000 tokens), with every system served over HTTP on the same node.我们回放 Mooncake 的 tool-and-agent trace<sup>[8](#ref-8)</sup>（prompt 平均约 9,000 个 token），所有系统都通过 HTTP 在同一个节点上提供服务。\n\nAt 1, 2 and 4 requests per second our engine reaches 100% SLO attainment, and 99.2% at 8; SGLang reaches 82 to 85% (a request meets its SLOs when its time to first token is at most 5 s and its mean time between tokens at most 0.1 s; attainment counted as in DistServe<sup>[9](#ref-9)</sup>). The gap comes from stalls: when a long prompt arrives, SGLang's decoding requests wait behind its prefill for up to 2.3 to 4.4 s; our megakernel's longest stall is 0.08 to 0.15 s.每秒 1、2 和 4 个请求时，我们的 engine 的 SLO 达成率是 100%，每秒 8 个时是 99.2%；SGLang 是 82 到 85%（首 token 时间不超过 5 s、token 间平均间隔不超过 0.1 s 即算满足，达成率按 DistServe<sup>[9](#ref-9)</sup> 的方式统计）。差距来自卡顿：长 prompt 到来时，SGLang 中正在 decode 的请求要排在它的 prefill 后面，最长等 2.3 到 4.4 s；我们的 megakernel 最长的卡顿是 0.08 到 0.15 s。\n\nOn Qwen3-8B and 1 × H100 the pattern is the same: SLO attainment is 95 to 97% versus 68 to 75% for SGLang and vLLM at 1 and 2 requests per second, and 90 to 93% versus 59 to 68% at 3; the longest stall is 0.09 to 0.13 s.在 Qwen3-8B 和 1 × H100 上规律相同：每秒 1 和 2 个请求时，SLO 达成率是 95 到 97%，SGLang 和 vLLM 是 68 到 75%；每秒 3 个时是 90 到 93%，对方是 59 到 68%；最长卡顿是 0.09 到 0.13 s。\n\n## How we validate an engine我们如何验证一个 engine\n\nFor autoresearch the **test suite** matters more than anything else: the agent optimizes whatever the evaluator rewards, so every gap in the evaluator becomes a shortcut. Agent-written kernels have passed tests by reusing the reference's output memory, by special-casing the test inputs and by timing the wrong stream[10](#ref-10)[11](#ref-11)<sup>[12](#ref-12)</sup>, and open kernel competitions now routinely filter out reward-hacked submissions<sup>[13](#ref-13)</sup>. The agent cannot modify the suite, and an engine is timed only after it passes.对 autoresearch 来说，**test suite** 比什么都重要：agent 优化的是 evaluator 奖励的东西，evaluator 的每个漏洞都会变成捷径。agent 写的 kernel 曾经靠复用 reference 的输出显存、针对测试输入写特例、计时测错 stream 而通过测试[10](#ref-10)[11](#ref-11)<sup>[12](#ref-12)</sup>，公开的 kernel 比赛现在也会例行过滤 reward hacking 的 submission<sup>[13](#ref-13)</sup>。agent 无法修改这套 suite，engine 通过之后才会被计时。\n\n**Reference and tolerances.** The reference is the model evaluated in FP64 (FP32 where FP64 does not fit in memory). Tolerances are calibrated on the standard BF16 implementation (Hugging Face Transformers<sup>[14](#ref-14)</sup>): we measure its deviation from the reference at every position and allow the engine twice that on the median and the 99th percentile of the logit error, as FlashAttention's own tests do<sup>[15](#ref-15)</sup>; four times that on the KL divergence of the next-token distribution, which tracks how many answers a change flips better than accuracy does<sup>[16](#ref-16)</sup>; and top-1 agreement within one point. vLLM and SGLang, measured the same way, deviate from the reference about as much as the BF16 implementation, so the tolerance is the error a general engine already has.**Reference 与容差。** reference 是用 FP64 计算的模型（显存放不下时用 FP32）。容差以标准的 BF16 实现（Hugging Face Transformers<sup>[14](#ref-14)</sup>）为基准：我们在每个位置测量它与 reference 的偏差，logit 误差的中位数和 99th percentile 允许 engine 达到它的两倍，与 FlashAttention 自己的测试做法相同<sup>[15](#ref-15)</sup>；next-token 分布的 KL divergence 允许四倍——它比 accuracy 更能反映一个改动会让多少答案翻转<sup>[16](#ref-16)</sup>；top-1 agreement 与它相差不超过一个百分点。按同样方式测量，vLLM 和 SGLang 与 reference 的偏差和 BF16 实现相当，所以这个容差就是通用 engine 本来就有的误差。\n\n**Inputs.** A public set covers every position up to 128 and every eighth position up to 1,152. A hidden set of natural text, code and math word problems extends to position 2,000; the run never sees it and only learns whether it passed.**输入。** 公开集覆盖 128 以内的每个位置，以及到 1,152 为止每隔八个的位置。一个由自然文本、代码和数学应用题组成的 hidden set 一直延伸到位置 2,000；run 看不到它，只知道自己是否通过。\n\n**The engine as it runs.** The engine's own decode loop must reproduce its step-by-step greedy tokens; the logits produced during the timed run are checked afterwards; and rerunning the batch with the sequences in reverse order must give bit-identical logits (in general engines a sequence's logits can depend on the batch it runs in<sup>[17](#ref-17)</sup>). Before release, each engine is also tested at every batch size, at full context and after a restart.**运行中的 engine。** engine 自己的 decode loop 必须复现它逐步 greedy 生成的 token；计时运行中产生的 logits 会在事后检查；把 batch 里的序列倒序后重跑，logits 必须逐位相同（在通用 engine 里，一个序列的 logits 可能取决于它所在的 batch<sup>[17](#ref-17)</sup>）。发布前，每个 engine 还要在每个 batch size、满 context 以及重启之后各测一遍。\n\n#### Table 2. What an engine has to pass表 2. engine 必须通过的检查\n\n| Check检查 | Compared with对比对象 | Passes when通过条件 | Runs运行时机 | \n|---|---|---|---|\n| Logits at every position每个位置的 logits | the model in FP64 (FP32 if it does not fit), on public and hidden sequencesFP64 计算的模型（放不下时用 FP32），公开序列和 hidden 序列 | median and 99th-percentile relative error ≤ 2× the BF16 implementation's相对误差的中位数和 99th percentile ≤ BF16 实现的 2× | every submission每次 submission | \n| Distribution分布 | the same reference同一个 reference | KL divergence median and p99 ≤ 4× the BF16 implementation's; top-1 agreement within one point of itKL divergence 的中位数和 p99 ≤ BF16 实现的 4×；top-1 agreement 与它相差不超过一个百分点 | every submission每次 submission | \n| Hidden setHidden set | natural text, code and word problems to position 2,000, never shown to the run自然文本、代码和应用题，到位置 2,000，从不给 run 看 | all of the above; only pass or fail is reported以上全部；只报告通过或失败 | every submission每次 submission | \n| As it runs运行中 | its own step-by-step greedy tokens它自己逐步 greedy 生成的 token | its decode loop gives the same tokens; logits taken during the timed run match the referencedecode loop 给出相同的 token；计时运行中取出的 logits 与 reference 吻合 | every submission每次 submission | \n| Slot order and repetitionSlot 顺序与重复 | the same batch with the sequences reversed序列倒序的同一个 batch | logits bit-identicallogits 逐位相同 | every submission每次 submission | \n| Batch sizes, full context, restartbatch size、满 context、重启 | fresh engines新启动的 engine | 1 to 8 sequences within the limits; bit-identical after a full context and after a restart1 到 8 个序列都在阈值内；满 context 之后和重启之后逐位相同 | before release发布前 | \n| Audit审计 | an independent agent reading the source一个独立的 agent 阅读源码 | no rule broken; its own experiments reproduce the scores没有违反任何规则；它自己的实验能复现分数 | before release发布前 | \n\nEvery engine shown in this post except the three Inkling-2-Small engines was re-tested with the current suite: 20 of 20 pass every check.除三个 Inkling-2-Small engine 外，本文展示的每个 engine 都用当前的 suite 重新测过：20 个中有 20 个通过全部检查。\n\n**Audits.** Every final engine was reviewed by an independent agent that read the source for rule violations and ran its own experiments. No engine calls another engine, caches outputs, special-cases the evaluator's inputs or skips part of the model. The audits did find problems that the evaluator of the time had passed; each led to a fix in the engine, in what we report, or in the suite.**审计。** 每个最终 engine 都由一个独立的 agent 审查：它阅读源码寻找违规之处，并跑自己的实验。没有 engine 调用别的 engine、缓存输出、针对 evaluator 的输入写特例或跳过模型的某一部分。但审计确实发现了当时的 evaluator 放过的问题；每一个都促成了一处修正：在 engine 里、在我们报告的内容里，或在 suite 里。\n\n## #### Table 3. What the audits found that the evaluator had passed表 3. 审计发现、而当时 evaluator 放过的问题\n\n| Finding发现 | Effect影响 | What was done处理 | Caught now by现在由什么检查拦住 | \n|---|---|---|---|\n| Two of the three timed prompts drive Qwen3.8-27B into a repetition loop that the drafts predict三个计时 prompt 中有两个会让 Qwen3.8-27B 陷入 draft 能预测的重复循环 | multi-token decoding scored 0.415 ms per token, 3.71 ms on natural textmulti-token decoding 得分 0.415 ms/token，在自然文本上是 3.71 ms | the natural-text figure is the one reported报告的是自然文本上的数字 | hidden natural-text set; natural-text timing自然文本 hidden set；自然文本计时 | \n| Three track-B engines labeled bit-identical were not三个标为逐位相同的 track-B engine 其实不是 | flint's engine picked a different top-1 token at 10.8% of positionsflint 的 engine 在 10.8% 的位置选了不同的 top-1 token | flint re-run, now bit-identical; the other two relabeledflint 重新跑过，现在逐位相同；另外两个改了标注 | bit-for-bit check against track A与 track A 的逐位比对 | \n| No error past the context limit超过 context 上限时不报错 | Inkling-2-Small overwrote other sequences' KV cache; flint returned empty outputInkling-2-Small 覆盖了其他序列的 KV cache；flint 返回空输出 | both runs extended; the engines refuse such requests两个 run 都延长了；engine 现在拒绝这类请求 | requests at and past the limit达到和超过上限的请求 | \n| A check added after the first Qwen3-8B run第一个 Qwen3-8B run 之后加的一项检查 | one of 1,024 positions over the tolerance: attention scores were rounded to BF16 twice1,024 个位置中有一个超出容差：attention score 被两次舍入到 BF16 | a 3-hour run fixed it at the same speed一个 3 小时的 run 修好了它，速度不变 | median and p99 instead of the maximum用中位数和 p99 代替最大值 | \n| Serving engines did not validate their inputsserving engine 不校验输入 | prefixes matched by a 64-bit hash without comparing tokens; no bounds checks前缀用 64-bit hash 匹配、不比较 token；没有越界检查 | Inkling-2-Small fixed in an extended run; Qwen3-8B being fixedInkling-2-Small 在延长的 run 中修好；Qwen3-8B 正在修 | mkserve input checks; collision testmkserve 输入检查；hash 碰撞测试 | \n| Qwen3-8B track-A serving under heavy load高负载下的 Qwen3-8B track-A serving | tokens differed from the same request run alone in 12 of 17 trials17 次试验中有 12 次，token 与同一请求单独运行时不同 | under investigation with the fix above与上面的修复一起排查中 | slot-order and load testsslot 顺序测试和负载测试 | \n\n**Timing.****计时。**\n\n- All systems of a comparison run in the same job on the same GPU, one after another.同一组对比的所有系统在同一个 job 里、同一块 GPU 上依次运行。\n- Three rounds; each restarts every server and begins with a different system.跑三轮；每轮重启所有 server，并由不同的系统先跑。\n- Every system is behind an OpenAI-compatible streaming endpoint and timed by the same client.每个系统都放在 OpenAI 兼容的 streaming endpoint 后面，由同一个 client 计时。\n- Prompts are token ids, and every request generates exactly 1,024 tokens.prompt 以 token id 给出，每个请求恰好生成 1,024 个 token。\n- vLLM and SGLang are released versions with their recommended flags, or a faster configuration where we found one.vLLM 和 SGLang 用发布版本和官方推荐的参数，我们找到更快配置的地方就用更快的。\n- An engine is timed only after it passes the correctness checks.engine 通过正确性检查之后才会被计时。\n\n## How the agents got thereagent 是怎么做到的\n\nA run gives Claude (Opus 5.5 or Fable 5.2, in Claude Code) the checkpoint, a GPU pod and a budget of agent-hours. The agent profiles and experiments as it sees fit and submits source code whenever it wants a score; each submission is logged with the hypothesis the agent wrote for it. The prompt states the goal, the rules and the tools, and nothing else. The engines in this post took 19 runs, 458 agent-hours and 675 submissions (every run is on the [data page](full.html#runs)).一个 run 给 Claude（Opus 5.5 或 Fable 5.2，在 Claude Code 中）一个 checkpoint、一个 GPU pod，以及一份 agent-hour 预算。agent 自行决定怎么 profile、怎么做实验，想要分数时随时提交源代码；每次 submission 都连同 agent 为它写下的 hypothesis 一起记录。prompt 只说明目标、规则和工具，别的什么都不给。本文中的 engine 用了 19 个 run、458 个 agent-hour 和 675 次 submission（每个 run 都在 [数据页](full.html#runs) 上）。\n\nThe runs follow a common trajectory. The first engine to pass the tests is slower than the fastest general engine. The first large gain comes from fusing the whole decoding step into one kernel and laying the weights out in the order the tensor cores consume them. Most later work targets the memory system: which loads go through which cache, what is prefetched in which phase, and what idle thread blocks do while attention runs. The engines overtake the general engines within about three hours on Qwen3-8B, North Mini Code and flint, and after seven and nineteen hours on Inkling-2-Small and Qwen3.8-27B. On Inkling-2-Small, a Fable 5.2 run starting from the Opus 5.5 run's engine at 5.5 hours surpassed, within four hours, what the Opus run reached in 24.各个 run 的轨迹相似。第一个通过测试的 engine 比最快的通用 engine 慢。第一次大的提升来自把整个 decoding step 融合进一个 kernel，并把权重按 tensor core 读取的顺序排布。之后的大部分工作针对内存系统：哪些 load 走哪一级 cache，哪个阶段预取什么，attention 运行时空闲的 thread block 做什么。在 Qwen3-8B、North Mini Code 和 flint 上，engine 大约三小时内就超过了通用 engine；在 Inkling-2-Small 和 Qwen3.8-27B 上分别用了七小时和十九小时。在 Inkling-2-Small 上，一个 Fable 5.2 run 从 Opus 5.5 run 第 5.5 小时的 engine 出发，四小时内就超过了 Opus run 用 24 小时达到的水平。\n\n## Limits and next局限与下一步\n\nThe steady-decode engines are narrow: at most 8 sequences and a 2,048-token context, the prompt consumed one token per pass, and sampling done outside the kernel (1 to 11% slower). Moving one setting at a time away from the benchmark on Inkling-2-Small: with 32 concurrent requests, SGLang delivers 3,338 tokens per second versus 1,427 for the steady-decode engine, which stays at its 8-sequence throughput; a 16,384-token generation or a 32,000-token prompt requires the serving engine, whose time to first token on a 32,000-token prompt is 0.70 s versus 0.41 s for SGLang. The serving engines handle long prompts and contexts, but under the production trace their tail time to first token and median time between tokens exceed SGLang's: on Inkling-2-Small at 1 to 8 requests per second, p99 time to first token is 3.4 to 4.6 s versus 2.0 to 2.2 s, and median time between tokens 28 to 60 ms versus 9 to 18 ms.steady-decode engine 的适用面很窄：最多 8 个序列和 2,048-token 的 context，每个 pass 只读入 prompt 的一个 token，sampling 在 kernel 外面做（慢 1 到 11%）。在 Inkling-2-Small 上每次只让一个设置偏离 benchmark：有 32 个并发请求时，SGLang 每秒输出 3,338 个 token，steady-decode engine 是 1,427，停在它 8 个序列时的吞吐；16,384-token 的生成或 32,000-token 的 prompt 都需要 serving engine，它在 32,000-token prompt 上的首 token 时间是 0.70 s，SGLang 是 0.41 s。serving engine 能处理长 prompt 和长 context，但在生产 trace 下，它的首 token 尾部时间和 token 间隔中位数都比 SGLang 长：Inkling-2-Small 在每秒 1 到 8 个请求时，首 token 时间的 p99 是 3.4 到 4.6 s，SGLang 是 2.0 到 2.2 s；token 间隔中位数是 28 到 60 ms，SGLang 是 9 到 18 ms。\n\nThe Qwen3-8B serving engines still match prompt prefixes by hash and rely on `mkserve` to reject invalid requests. All numbers come from single nodes and two to three rounds.Qwen3-8B 的 serving engine 目前仍用 hash 匹配 prompt 前缀，并依赖 `mkserve` 拒绝无效请求。所有数字都来自单个节点，跑了两到三轮。\n\n**Next.** Anyone can run this autoresearch with their own agent on the model they care about: [megakernel-zoo](https://github.com/dangxingyu/megakernel-zoo) has the test suite and the tools to start a run on a new checkpoint, and an engine that passes the suite can be contributed back to the zoo. Over time, whenever someone wants to run a model, the fastest megakernel for it and their GPU can already be there, on demand. On our side, the next step is a serving engine whose small passes are as fast as the steady-decode kernel, so that one engine does both jobs. That engine needs the steady-decode speed while one to eight requests are decoding, prompt chunks fast enough that under load the first token and the gaps between tokens are at SGLang's level or better, and sampling inside the kernel.**下一步。** 任何人都可以用自己的 agent，在自己关心的模型上跑这套 autoresearch：[megakernel-zoo](https://github.com/dangxingyu/megakernel-zoo) 提供了 test suite 和在新 checkpoint 上启动 run 的工具，通过 suite 的 engine 可以贡献回 zoo。久而久之，有人想跑某个模型时，适合这个模型和他那块 GPU 的最快 megakernel 可能已经在那里，随取随用。我们这边的下一步，是做一个小 pass 和 steady-decode kernel 一样快的 serving engine，让一个 engine 同时胜任两件事。这个 engine 需要：一到八个请求在 decode 时达到 steady-decode 的速度；prompt chunk 处理得足够快，使负载下第一个 token 的时间和 token 之间的间隔不差于 SGLang；以及在 kernel 内部做 sampling。\n\n## Acknowledgements致谢\n\nWe thank Wentao Guo for sharing intuitions and a philosophy of megakernel design and for helpful discussions and suggestions, and Wenhao Chai for suggestions on the presentation.感谢 Wentao Guo 分享对 megakernel 设计的直觉和理念，以及有益的讨论和建议；感谢 Wenhao Chai 对呈现方式的建议。\n\n## References\n\n1. **1** Kwon et al.[Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180) . SOSP 2023.\n2. **2** Zheng et al.[SGLang: Efficient Execution of Structured Language Model Programs](https://arxiv.org/abs/2312.07104) . NeurIPS 2024.\n3. **3** Spector, Juravsky, Sul, Dugan, Lim, Fu, Arora and Ré.[Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B](https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles) . Hazy Research blog, 2025.\n4. **4** Tong, Zhu and Wang.[Inside the megakernel serving engine for North Mini Code](https://cohere.com/blog/megakernels) . Cohere blog, 2026.\n5. **5** Cheng et al.[Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs](https://arxiv.org/abs/2512.22219) . arXiv 2025.\n6. **6**\n7. **7** Li et al.[SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding](https://arxiv.org/abs/2611.02501) . arXiv 2026.\n8. **8** Qin et al.[Mooncake: Trading More Storage for Less Computation, a KVCache-centric Architecture for Serving LLM Chatbot](https://www.usenix.org/conference/fast25/presentation/qin) . FAST 2025. Trace:[FAST25-release/traces](https://github.com/kvcache-ai/Mooncake/tree/main/FAST25-release/traces) .\n9. **9** Zhong et al.[DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://arxiv.org/abs/2401.09670) . OSDI 2024.\n10. **10** Ouyang et al.[KernelBench: Can LLMs Write Efficient GPU Kernels?](https://arxiv.org/abs/2502.10517) ICML 2025.\n11. **11** Lange et al.[The AI CUDA Engineer: Agentic CUDA Kernel Discovery, Optimization and Composition](https://sakana.ai/ai-cuda-engineer/) . Sakana AI, 2025, and its correction of February 2025.\n12. **12** Zhang et al.[KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch?](https://arxiv.org/abs/2607.16241) arXiv 2026.\n13. **13** Baslak, Schultheis, Sirovatka, Zhang and Saroufim.[We Let Agents Compete and They Tried to Cheat. KernelGuard: Defending GPU Competitions from Adversarial Agentic Systems](https://icml.cc/virtual/2026/67922) . ICML 2026.\n14. **14** Wolf et al.[Transformers: State-of-the-Art Natural Language Processing](https://arxiv.org/abs/1910.03771) . EMNLP 2020 (System Demonstrations).\n15. **15** Dao.[FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://arxiv.org/abs/2307.08691) . ICLR 2024. Tolerance rule in[tests/test_flash_attn.py](https://github.com/Dao-AILab/flash-attention/blob/main/tests/test_flash_attn.py) .\n16. **16** Dutta, Krishnan, Kwatra and Ramjee.[Accuracy is Not All You Need](https://arxiv.org/abs/2407.09141) . NeurIPS 2024.\n17. **17** He and Thinking Machines Lab.[Defeating Nondeterminism in LLM Inference](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/) . Connectionism, 2025.", "url": "https://wpnews.pro/news/a-megakernel-per-model", "canonical_source": "https://dangxingyu.github.io/megakernel/", "published_at": "2026-11-19 12:08:12+00:00", "updated_at": "2026-11-19 12:46:37.937569+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-agents", "mlops", "ai-tools"], "entities": ["vLLM", "SGLang", "Inkling-2-Small", "B200", "Claude Opus 5.5", "Fable 5.2", "megakernel-zoo", "Hazy Research"], "also_reported_by": [], "takeaway": "Agent-written per-checkpoint megakernels beat the fastest general engine by 1.13–1.77× on single-sequence decode across eight models from 8B to 594B parameters, reaching 2.05× when model storage can change without altering numerical output.", "recommendation": {"advice": "Fuse the entire decode loop into one megakernel per checkpoint and GPU so operators can be rescheduled and weights prefetched across layer boundaries; the source measured 1.13–1.77× faster single-sequence decode than vLLM or SGLang.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/a-megakernel-per-model", "markdown": "https://wpnews.pro/news/a-megakernel-per-model.md", "text": "https://wpnews.pro/news/a-megakernel-per-model.txt", "jsonld": "https://wpnews.pro/news/a-megakernel-per-model.jsonld"}}