{"slug": "what-a-kernel-is-and-why-everyone-is-writing-new-ones", "title": "What a Kernel Is, and Why Everyone Is Writing New Ones", "summary": "A kernel is a single function that runs on a GPU, and the steep memory hierarchy—with a 1,600x gap between L2 cache and HBM—makes kernel design crucial for AI performance. FlashAttention, developed by researchers including Tri Dao, avoids materializing the full attention matrix by processing in tiles and using online softmax, which is why it achieves large speedups on decoder models. The article is part three of a seven-part series explaining the layer where most AI vocabulary originates.", "body_md": "Part three of seven. One layer above the silicon, and the source of most of the names you have heard.\n\nPart one established that the GPU can calculate far faster than it can fetch, and part two showed how much there is to fetch. This part is about the layer where somebody actually does something about it.\n\nIt is also where nearly all the vocabulary comes from. FlashAttention, FlashInfer, Triton, CUTLASS, CUDA: every one of those is a name in this single layer, and they make far more sense once you know what the layer does.\n\nA **kernel** is a single function that runs on the GPU. You launch it, thousands of threads execute it in parallel, and it finishes. Multiplying two matrices is a kernel. So is applying softmax, the step that turns a row of raw scores into fractions that add up to one. Running a whole model means launching hundreds of them in sequence. CUDA is NVIDIA’s platform for writing them, and everything else named above is a library or a language sitting on top of it.\n\nThe word sounds harder than it is. What makes kernels interesting is the memory hierarchy they sit in, and how steep it is:\n\n*Drawn to scale, because most of the other descriptions of this are wrong. It is not a smooth ladder.*\n\nThe shape is not what people describe. Three of those tiers hold roughly the same amount: 33 MiB, 29 MiB and 50 MB. The register file is actually larger than the shared memory below it, so the tiers do not even get bigger as you go down. There is one real step, and it sits between L2 cache and HBM: **1,600 times**.\n\n**Shared memory** is the tier that matters here: a scratchpad physically on the chip, an order of magnitude quicker to reach than main memory and shared by the threads of one block. The catch is that there is almost none of it. An SM (streaming multiprocessor) is one of the chip’s independent compute units, and each one gets 228 KiB. A single block gets only 48 KiB of that unless the kernel explicitly opts in for more.\n\nSo a good kernel is not one that does clever arithmetic, it is one that gets data into that scratchpad and wrings as much work out of it as it can before going back for more.\n\nThe obvious way to compute attention is to follow the equation. For a sequence of length S, you compute a score between every token and every other token, which is an S by S matrix. Write it to memory. Read it back to apply softmax. Write that. Read it again to multiply by the values.\n\nFour trips across the slowest boundary in the machine, for a matrix that grows with the square of your sequence length. At 64K tokens that matrix is 8.6 GB, for one query head in one layer.\n\n*At 64K tokens the score matrix is 8.6 GB, for one query head in one layer. gpt-oss-120b has 64 query heads, and 18 layers that attend to the whole sequence. Causal masking halves the arithmetic, since a token never attends to its future, but it does not halve that 8.6 GB: a naive implementation still allocates the whole square and fills half of it with negative infinity. FlashAttention has no matrix to allocate, so it skips the fully masked tiles altogether, and that is where a good share of the speedup on decoder models comes from.*\n\nThat is the number that turned this layer into a research field.\n\nIt never writes the score matrix down. Not once.\n\nInstead it processes the sequence in **tiles**: load a block of queries and a block of keys into shared memory, compute their scores there, use them, discard them, move on. The full matrix never exists at all. Only the tile being worked on does, briefly, on the chip.\n\nThe trick that makes this possible is **online softmax**, and it came before FlashAttention. Milakov and Gimelshein described it in 2018, and in 2021 Rabe and Staats showed attention could be computed without ever materializing the full matrix. Softmax normally needs to see every score before it can normalize, which is exactly why the naive version has to store them all. The online version keeps a running maximum and a running sum, and rescales its partial result each time a new block arrives. When the last block has been processed, the answer is correct — exactly correct, not nearly.\n\n*Same tiles, same order, same answer. The gold outline is the tile being computed. The left keeps every one it has finished; the right keeps whichever it is holding and drops the rest, which is why one counter climbs and the other does not.*\n\nThis gets described loosely, and two different claims get merged into one, so it is worth being precise.\n\n**It computes exact attention.** It does not approximate. The “efficient-attention” papers it gets filed alongside all go faster by dropping part of the calculation or compressing it. This one drops nothing. It is *not* bit-identical to the naive version, because summing in a different order changes the last few bits of a floating-point result. In practice it lands closer to a double-precision reference than the naive version does, since it accumulates in 32-bit.\n\n**And on the tiles it does compute, it does more arithmetic, not less.** Streaming a tile at a time means the number softmax has to divide by is not known until the last tile arrives, so the running total has to be rescaled as each new tile arrives. That rescaling is the extra work in the forward pass. In training there is a second helping of it, where the backward pass recomputes the scores from Q, K and V rather than having stashed them. Either way the trade is the same: spend FLOPs, save bytes, on hardware where bytes are the scarce thing.\n\nHeld at the peak, that is 8.6 GB against 34 MB, a factor of about 250. The saving is in what is *held*, which is a different question from what is *moved*:\n\nThe traffic win is the smaller of the two, and it does not depend on sequence length at all. The ratio is twice the query-block size over the head dimension: 256/64 here, so a factor of four. It is a property of how the kernel tiles, not of how long your prompt is, and the work still grows with the square of the sequence either way, because each block of queries re-reads the keys and values. The *footprint* is where the order-of-magnitude change lives: a matrix that could not be allocated at 64K simply stops existing.\n\nThat is the lesson of this whole layer, and it is easy to state it wrongly. Part one called prefill the comfortable case: far to the right of the ridge, compute-bound. Both things are true, and the order matters. Naive attention was bandwidth-bound**, and FlashAttention is what moved it back to being compute-bound.** That is why the newest versions of it chase how busy they keep the tensor cores, rather than bytes. The round trips were the expensive part until somebody removed them.\n\nThis one causes real confusion, so here they are side by side.\n\n**Triton, the language**, is a way of writing GPU kernels in something that looks like Python. You express the work in blocks of data rather than individual threads, and the compiler handles the placement and scheduling *within* a block. Not all of it: you still choose how big the blocks are, how many threads run in lockstep, and how deep the pipeline goes. Searching for the best combination of those is part of the job rather than a finishing touch. It is roughly a tenth of the code of the traditional approach, for most of the performance. It is also not an NVIDIA project: it started at Harvard and OpenAI, and it now targets AMD and Intel too.\n\nPyTorch’s compiler emits it, which is the part most people are unaware of, but selectively: torch.compile generates Triton for the element-by-element and summing work on GPU, while matrix multiplications still go to cuBLAS unless you ask for max-autotune. If you want to see what it produced, TORCH_LOGS=output_code prints it.\n\n**Dynamo Triton is an unrelated product for serving models over a network. NVIDIA renamed Triton Inference Server to it in March 2025, when the server folded into the Dynamo platform.** Different origin, different problem, different layer: it lives at the top of this stack, not the bottom. The rename is slowly clearing that up, but a lot of writing came before it, so both names will confuse people for years yet.\n\nThey share a name and nothing else. Say “Triton kernels” or “Triton server” and never just Triton. It saves twenty minutes of talking past each other.\n\nThe other name you will meet is **CUTLASS**, NVIDIA’s template library for the operations underneath all of this: GEMM, which is a matrix multiply, and its batched and grouped variants. Since version 3 it also carries CuTe, the layout system that FlashAttention-3 is built on. On Hopper and Blackwell it is not what you reach for after hand-tuning. It is how you reach the hardware’s newest paths at all. The case that matters most for this series is grouped GEMM, which is what makes a mixture-of-experts layer fast, and there the gap over a naive implementation is far more than a last ten percent.\n\n**FlashInfer** is a library of attention and matrix kernels built specifically for serving. It is the default in SGLang, and in vLLM it is first choice on Blackwell. It is not first choice on the H100 this series has been computing on, where vLLM still picks FlashAttention and puts FlashInfer second. The direction also gets reported backwards: TensorRT-LLM does not sit on top of FlashInfer, it ships kernels *into* it, alongside FlashAttention, cuDNN and CUTLASS.\n\nSo the honest version is you may well be running it, and which one you get depends on your GPU generation and your engine. I will discuss how to check in the next section. Two ideas in it are the ones to know.\n\n**Every cache layout is treated as one kind of sparse matrix.** Paged cache, shared prefixes, sliding windows: rather than writing a separate kernel for each, FlashInfer represents them all in a single block-sparse format and generates the kernel from a template. One implementation covers layouts that would otherwise each need their own.\n\n**It plans before it runs.** Real serving batches are ragged, with different prompt lengths and different positions in generation. A fixed launch shape wastes capacity on whichever request is smallest. So FlashInfer inspects the batch first, works out a balanced schedule, and only then launches. In a real serving loop the batch changes every step, so the plan is rebuilt every step. It spreads that cost over the 36 layers of a single forward pass, and it runs on the CPU while the GPU is still busy.\n\n*The whole vocabulary of this layer, in the order you are likely to meet it. The bracket marks the only pair that shares a name, which is the one worth keeping straight.*\n\nYou will almost certainly not write a kernel. All of this assumes NVIDIA on Linux. Windows means WSL2, so the same stack underneath; on Apple Silicon none of these names exist, because the Metal plugin runs MLX kernels instead. What you will do is find out which ones you are running, and it helps to know in advance how much that is worth. Attention is about 18% of prefill time at 4K and 85% at 128K, so which regime you are in decides whether a faster kernel is a rounding error or the whole game. Part seven puts a number on both ends.\n\n**Read the backend line.** Your engine logs which attention backend it chose at startup, and you can force it, with the same flag name but different values in vLLM and SGLang: --attention-backend FLASH_ATTN. The list of names is the best argument for this article existing, because TRITON_ATTN and CUTLASS_MLA are both in it. The two libraries above are not background reading. They are options in your own log line.\n\nWhich one you get is decided by your GPU generation. On Hopper vLLM ranks FlashAttention first and FlashInfer second; on Blackwell it reverses them. And FlashAttention itself has versions: FA2, then FA3 for Hopper’s newer instructions, then FA4 on Blackwell. So “FlashAttention” in a log line is half an answer.\n\n**Know what silently drops you off the fast path.** Not usually some unusual head dimension. In practice it is FlashInfer not being installed, logits soft-capping, an FP8 KV cache, or a sliding-window pattern the backend does not support. The model in part two uses that pattern on half its layers. Nothing fails. It is just slow, indefinitely.\n\n**And check CUDA graphs are on**, because this is the larger and quieter cliff. Each kernel launch costs microseconds, a decode step is hundreds of launches, and against part one’s 21 ms step that adds up. vLLM captures the whole step as a graph to avoid it. Running with --enforce-eager disables that. It belongs in development and nowhere else.\n\nOne caveat on the 8.6 GB, since the whole article is built on it. Chunked prefill is on by default, so a 64K prompt is not processed as one 64K-token square. What chunking bounds is the *query* side: a couple of thousand queries at a time. Each of those chunks still attends to every key before it, so the last chunk of that prompt computes a 2,000 × 64K block, not a 2,000 × 2,000 one. The total is unchanged; it is paid in slices. That is why the number still matters, and why the growth is what made anyone build FlashAttention in the first place.\n\nIf your engine offers a newer backend behind a flag, benchmark it on your own request shapes rather than trusting the release notes. vllm bench serve exists for exactly that.\n\nAnd when somebody shows you a kernel benchmark, remember what it is measuring: **one operation, in isolation, on fixed inputs**.\n\nEverything in this layer exists because of one observation.\n\nThe arithmetic was never the expensive part. It was the round trips.\n\nFlashAttention computes exact attention, and it does so by doing *more* arithmetic rather than less. It changed the field anyway, because it stopped an 8.6 GB matrix from ever having to exist. That is the shape of almost every kernel result you will read about, and it is the thing to hold onto when the next one is announced.\n\nPart four is quantization: what it means to store the model’s numbers in fewer bits, and how to read the accuracy claim that comes with it.\n\n*Memory-traffic figures use gpt-oss-120b’s geometry, head dimension 64 at 16-bit, and a query block of 128. That block size is where the 256 in the ratio comes from, and it is a property of the kernel rather than the model.*\n\n[What a Kernel Is, and Why Everyone Is Writing New Ones](https://pub.towardsai.net/what-a-kernel-is-and-why-everyone-is-writing-new-ones-112411ef5c36) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/what-a-kernel-is-and-why-everyone-is-writing-new-ones", "canonical_source": "https://pub.towardsai.net/what-a-kernel-is-and-why-everyone-is-writing-new-ones-112411ef5c36?source=rss----98111c9905da---4", "published_at": "2026-08-18 21:01:01+00:00", "updated_at": "2026-08-18 21:42:07.626747+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-infrastructure"], "entities": ["NVIDIA", "CUDA", "FlashAttention", "FlashInfer", "Triton", "CUTLASS", "Milakov", "Gimelshein"], "alternates": {"html": "https://wpnews.pro/news/what-a-kernel-is-and-why-everyone-is-writing-new-ones", "markdown": "https://wpnews.pro/news/what-a-kernel-is-and-why-everyone-is-writing-new-ones.md", "text": "https://wpnews.pro/news/what-a-kernel-is-and-why-everyone-is-writing-new-ones.txt", "jsonld": "https://wpnews.pro/news/what-a-kernel-is-and-why-everyone-is-writing-new-ones.jsonld"}}