An AITER Pin Keyed by Kernel Text: Faster Glue Than the Inductor Default An offline kernel-tuning approach that pins Triton glue-kernel configs keyed by kernel text delivers 1.388x speedup on Model A and 2.846x on Model B over TorchInductor's single default config, and 1.15x on Model B over Inductor's open-list autotune, according to the post's measurements on two unnamed production models. The method pulls the search out of the Inductor compile, keeping a config only when a numerical check passes and the trace is faster, then writes it into the shipped package so startup does not time candidates. The post reports that Model A's open-list autotune compiles produced unstable winners of 1618.9, 1769.8 and 1863.5 us — a 245 us spread — larger than Model A's edge over its fastest autotune compile, so that gain is not counted as a listed speedup. An AITER Pin Keyed by Kernel Text: Faster Glue Than the Inductor Default an-aiter-pin-keyed-by-kernel-text-faster-glue-than-the-inductor-default GEMM and attention kernels already have offline tuning paths. The kernels around them often do not. A compiled inference graph still launches a long tail of small Triton kernels for casts, concatenations, normalizations, activations, gathers, and histogram-style updates. TorchInductor generates those kernels. With autotune on, it times a candidate list and writes the winner into a cache keyed by a fusion number. Figure 1, below, is why that cache is not what ships. Two limits of that flag are why the search is pulled out of the compile. It is a closed choice: it times the list the compiler emits, and the winner it stores is the config you get. A faster tile that the list does not contain is not a choice it can make. Another process on the GPU can make the fastest config in that list look slow. Model A’s three open-list compiles were not screened for a second process on the GPU. What they measured is an unstable winner: 1618.9, 1769.8, and 1863.5 us, a spread of 245 us. The offline search runs apart from that compile, keeps a config only when the check passes and the trace is faster, and writes it into the package. Startup does not time candidates. On glue the kernel speedup against Inductor’s single default config is 1.388x on Model A and 2.846x on Model B. Against the open-list autotune, Model B is 1.15x. Model A’s edge over its fastest autotune compile is smaller than the 245 us spread, so it is not a listed speedup. Table 1 and Table 9 hold the times. That tail is the glue in this post: pointwise, reduction, and persistent-reduction kernels. The tables compare the three stages. - Inductor keeps the TorchInductor body and turns glue autotune off, so every glue kernel launches the compiler’s single default config. - AITER replaces that body with another Triton function that has the same arguments, and leaves the config at the Inductor default. AITER here is a library of Triton kernels for AMD Instinct GPUs. A body that fails the numerical check stays on the compiler source. - Tuned is the package that ships. On Model A, 12 kept configs are in the compiler list and 55 are not. Forty-six of those 55 change a block, warp, or stage, counted in the stage definition below. In the tables, the AITER row is that body swap at the default config. The AITER pin in the title is the Tuned config file. A fourth build is a control, not a stage. It puts that file back on the Inductor body, which is the last row of Figure 2. Two unnamed production models are the measurements. Model A has 143 glue kernels and Model B has 25. GEMM and attention template timings are copied across builds. The glue autotune cache is not, so a glue change is not a GEMM being selected again. The GEMM binary stays the same, though its duration can still move. Request latency, profiler off, is the end-to-end number. In this blog, you will learn why that search leaves the Inductor compile, how a pin is keyed by the kernel text, and what the traces say about a body swap and a config change. Where This Is Worth Doing where-this-is-worth-doing The shape is fixed, so a measured config is the one production launches. A unit in the last place ULP is one step of a bf16 or fp16 value. A kernel that fails the check stays on the compiler body. This does not help when the hot path is still an untuned GEMM, the shape changes every step, or the time is a launch the body cannot fuse away. What the Glue Kernels Are what-the-glue-kernels-are TorchInductor fuses elementwise work, small reductions, and indexing into Triton so the graph does not bounce through global memory between ops. A pointwise kernel owns XBLOCK elements. A reduction loops over RBLOCK and then collapses it. A persistent reduction keeps that axis in one tile. Warp count, stages, and waves per eu sit beside the tile. A wave on this GPU is 64 threads. waves per eu is how many of those waves the launch asks to keep resident on one execution unit. From a Compiler Kernel to a Pinned Triton Kernel from-a-compiler-kernel-to-a-pinned-triton-kernel The conversion does not rewrite the graph. The compiled graph still calls one function per fusion, with the same pointers and the same grid. Two things about that function are allowed to change: the launch config, and the statements inside the @triton.jit function. Capture capture Compile the graph once the way production compiles it, and keep the generated Python for every glue kernel. Each file is a heuristic decorator plus one @triton.jit function. The decorator records the signature, the size hints, and the configs autotune is allowed to try. The function is the body. Kernels are matched by a fingerprint: the SHA-256 of the @triton.jit function with the numeric suffix removed, truncated to 20 hex characters. Inductor renumbers fusions when an upstream template changes, so the suffix is not an identity. When two pins share a fingerprint and disagree, a pin whose kernel name still matches is applied by name. The artifact that ships is an AOTInductor package plus the runtime graph. The build uses one compile thread, so the patch is visible to the compiler, and the log records how many pins and bodies were applied. The Three Stages the-three-stages Figure 2, below, changes the body and the config one at a time. Doing both in one step would credit a tile to the body, or a body to the tile. The control then puts the pins back on the compiler body. The Inductor stage turns glue autotune off and keeps the compiler body. On Model A, max-autotune still offers several candidates, so the stage slices that list to the first config. On Model B the flag alone yields one config, and that build is the Inductor package below. Every later ratio divides by this stage. The AITER stage swaps the @triton.jit source and leaves the signature, the decorator, and the arguments alone. On Model A the swap is three SiLU kernels, a sigmoid a b , out of 143. On Model B, 24 of 25 files match the compiler text and the 25th differs only in constants the build reinitializes. The one-config slice stays. An earlier Model B install left the list open and moved the tile, so the AITER row is a rebuild that cuts the list to one config first. The open-list build is reported beside it. Each kernel is screened for three rounds of five repetitions. The three fastest that pass are confirmed for seven rounds of ten, interleaved with the baseline. A candidate is kept when its median is at least 0.5 percent faster and its maximum over those seven rounds sits below the baseline’s minimum. That interval is the min and the max, not a confidence interval. Inside that noise, the grid that can fill the 80 compute units wins. A kernel the sweep cannot improve keeps the config it arrived with. On Model A, pointwise candidates use XBLOCK from 64 up to the power of two covering the launch, warps in {1, 2, 4, 8}, waves per eu in {0, 2}, and one stage. Reduction candidates use R0 BLOCK from 64 to 4096, the same warps, and one or two stages. A tile that would put more than 64 elements on a thread is dropped. Of the 143, 69 are pointwise, 64 are persistent reductions, and 10 are reductions. A pointwise kernel that also has a YBLOCK axis is searched on both tiles together, with the product of the two blocks kept between 256 and 16384 elements. Across the 143 that is 5220 candidates. 1329 failed the bitwise check: 119 pointwise, 789 persistent reduction, and 421 reduction. None failed to compile. 67 kernels took a new config, and the other 76 kept the baseline. Of the 67, 12 match a config the compiler heuristic emits for that kernel’s size hints on the open-list compile. The other 55 do not. Nine of the 55 keep the block size, warp count, and stage count and set waves per eu to 2. The other 46 change a block size, the warp count, or the stage count. Those 46 are the wider part of the search. Model B refuses a tile larger than the config in hand, so the XBLOCK 128 already chosen by the compiler list stayed the baseline. The 119 pointwise failures are five kernels, none of them atomic and none a reduction, split 45, 45, 17, 8, and 4. The first three are a masked split across inputs and a divide. The other two are masked pointwise kernels as well: 38 of 46 candidates passed on one, and 41 of 45 on the other. The remaining 64 pointwise kernels had no gate failures. On Model B, XBLOCK is 1 or a power of two from 8 to 4096, warps are {1, 2, 4, 8, 16}, stages are 1 to 4, and waves per eu is {0, 2, 4, 8}. The tile cannot exceed the config in hand, and the grid stays under about a thousand waves per compute unit. Of the 25 kernels, 17 took a new config, 6 kept the previous one because nothing cleared 0.5 percent, and 2 had no passing candidate. A fast sigmoid, allowed one ULP of bf16 error, made glue slower on both models and was not kept. A reciprocal hoist that leaves the stored bits unchanged is in the ceiling section. It is not in the Tuned package. The rule used to accept or reject a body is the same one used for the config sweep. - Integer outputs stay equal. - bf16 and fp16 may move by at most one ULP when the change is an explicit fast-math form such as the sigmoid. Everything else stays bitwise. - fp32 stays bitwise. Atomics are compared after the outputs are zeroed, because addition order changes the low bits. Buffers that are also inputs are left intact. Clearing those inputs caused false rejects. A body that fails is dropped. The compiler’s version of that kernel stays. How the Traces Were Read how-the-traces-were-read Kernel time comes from rocprofv3 kernel traces. The benchmark’s duration field ends the run, because the tool writes the file only when the process exits on its own. rocprofv3 --kernel-trace --output-format csv -d prof -o trace --