A September 29, 2026 arXiv paper reports that an LLM agent translating Triton kernels directly into NVIDIA PTX, a process the authors call "AI lowering," achieved 0.83x to 3.34x the performance of aut…
A developer who built InsNet, a ~21,000-line C++14 deep learning library with 4,828 lines of hand-written CUDA kernels, starting in 2018, describes how its padding-free dynamic batching design anticip…
RBS-Attention, a training-free sparse-prefill method from a new arXiv paper (2609.20971v1), achieves 20.65x standalone prefill-attention speedup, 11.92x vLLM prefill-attention speedup, and 5.97x end-t…
Researchers introduced Simple Attention Sparsification (SAS), a gated sparse attention mechanism that optimizes context ranking end-to-end with the language modeling loss by injecting the selector's c…
A technical handbook on FlashAttention explains that the algorithm achieves wall-clock speedups by reducing data movement between GPU high-bandwidth memory (HBM) and on-chip SRAM rather than by approx…
Manas Pathak's primer 'What Happens Inside an LLM' explains the internal workings of a large language model during a forward pass, detailing how tokens are converted into vectors, processed through tr…
A developer documented writing GPU kernels in Triton to understand PyTorch operations, starting with vector add and fused ReLU/dropout, highlighting the benefit of fused kernels in reducing memory rou…
A developer's source-code walkthrough of vLLM 0.22's V1 execution path traces a single offline inference request from the LLM.generate() API through inter-process communication, scheduling, GPU execut…
Developer Maneesh Kumar Thakur built a map of 149 generative AI tools sorted by concept depth rather than hype, distinguishing ease of use from the understanding required to use each tool effectively.…
A kernel is a single function that runs on a GPU, and the steep memory hierarchy—with a 1,600x gap between L2 cache and HBM—makes kernel design crucial for AI performance. FlashAttention, developed by…
The KV cache has no standard ABI, with vLLM's FlashAttention backend alone reporting its cache shape as a four-dimensional tensor that varies by backend, attention variant, and model family, complicat…
A new critique argues that the term 'linear attention' is overused and ambiguous, covering at least six different resource claims that range from constant computation per token to merely reduced KV ca…
AMD introduced VSA (Video Sparse Attention), a hardware-efficient sparse attention mechanism implemented with CK Tile, achieving a 3.31× attention kernel-time speedup at 70% sparsity over FlashAttenti…
NVIDIA has published a technical post analyzing how group size, head dimension, and sequence length affect dense attention performance in long-context inference, offering a co-design checklist to help…
HexCore, a low-latency paged KV cache allocator for LLM inference written in C++20 and CUDA, has been released under the Apache License 2.0 by Rasuljanov Muhammadali. The CPU-side allocator and relate…
Astral has launched pre-built GPU-enabled Python wheels for the PyTorch ecosystem, supporting packages like FlashAttention across multiple CUDA versions (11.8 through 13.2) and PyTorch versions. The A…
Microsoft's Mage-Flow image generation model, with only 4 billion parameters, outperforms larger models including FLUX.2-dev (32B) and Qwen-Image (20B) on GenEval benchmarks, scoring 0.88 against thei…
Rotary Position Embedding (RoPE), introduced by Su et al. in 2021, solves the Transformer long-context problem by rotating Query and Key vectors in 2D sub-planes, making attention depend only on relat…
PyTorch 2.13, released July 8, brings FlexAttention to Apple Silicon with up to 12x speedup on sparse attention patterns, such as a 32,768-token sequence with a 256-token sliding window (35ms vs 431ms…
Researchers have introduced STEEL, the first open-source implementation of FlashAttention optimized for XDNA-like neural processing units (NPUs), achieving significant energy and speed advantages for …