A kernel is a single function that runs on a GPU, and the steep memory hierarchy—with a 1,600x gap between L2 cache and HBM—makes kernel design crucial for AI performance. FlashAttention, developed by…
The KV cache has no standard ABI, with vLLM's FlashAttention backend alone reporting its cache shape as a four-dimensional tensor that varies by backend, attention variant, and model family, complicat…
A new critique argues that the term 'linear attention' is overused and ambiguous, covering at least six different resource claims that range from constant computation per token to merely reduced KV ca…
AMD introduced VSA (Video Sparse Attention), a hardware-efficient sparse attention mechanism implemented with CK Tile, achieving a 3.31× attention kernel-time speedup at 70% sparsity over FlashAttenti…
NVIDIA has published a technical post analyzing how group size, head dimension, and sequence length affect dense attention performance in long-context inference, offering a co-design checklist to help…
HexCore, a low-latency paged KV cache allocator for LLM inference written in C++20 and CUDA, has been released under the Apache License 2.0 by Rasuljanov Muhammadali. The CPU-side allocator and relate…
Astral has launched pre-built GPU-enabled Python wheels for the PyTorch ecosystem, supporting packages like FlashAttention across multiple CUDA versions (11.8 through 13.2) and PyTorch versions. The A…
Microsoft's Mage-Flow image generation model, with only 4 billion parameters, outperforms larger models including FLUX.2-dev (32B) and Qwen-Image (20B) on GenEval benchmarks, scoring 0.88 against thei…
Rotary Position Embedding (RoPE), introduced by Su et al. in 2021, solves the Transformer long-context problem by rotating Query and Key vectors in 2D sub-planes, making attention depend only on relat…
PyTorch 2.13, released July 8, brings FlexAttention to Apple Silicon with up to 12x speedup on sparse attention patterns, such as a 32,768-token sequence with a 256-token sliding window (35ms vs 431ms…
Researchers have introduced STEEL, the first open-source implementation of FlashAttention optimized for XDNA-like neural processing units (NPUs), achieving significant energy and speed advantages for …
Developer Vincenzo released NanoEuler, a from-scratch GPT-2-scale language model training stack written in C and CUDA without PyTorch or autograd, on June 28, 2026. The 116-million-parameter project i…
A new book, 'Modern GPU Programming For MLSys', teaches GPU kernel optimization for machine learning systems, focusing on Blackwell architecture and techniques like GEMM and FlashAttention. Developed …
A developer released NanoEuler, a GPT-2-scale language model built entirely from scratch in C/CUDA without any machine learning libraries. The project includes a hand-written byte-level BPE tokenizer,…
Miami startup Subquadratic claims to have solved a decade-old bottleneck in AI models by replacing dense attention with sparse attention, achieving 56x speed gains and dramatic cost reductions. Indepe…
VLLM, a model-serving engine for large language models, introduced a small op-level IR to resolve the tension between acting as a compiler target and a hand-tuned kernel dispatcher. The IR allows vLLM…
A blog post derives the backward pass of attention mechanisms step by step, using a concrete example to illustrate gradient computation for Q, K, and V matrices. The derivation builds on FlashAttentio…
Shrijith Venkatramana, developer of git-lrc, explains how kernel fusion reduces memory bandwidth bottlenecks in LLM inference. By combining multiple GPU operations into a single kernel, intermediate d…
A new dataset, fineset-io/efficient-llm-papers, compiles 1,734 records of arXiv and Semantic Scholar papers on efficient LLM techniques like quantization, LoRA, MoE, and FlashAttention, each quality-s…
Rudrite Research launched a free, open platform offering interactive, animated visual explainers of landmark AI and ML papers, including Attention Is All You Need, GPT-3, and FlashAttention, to make f…