SGLang and vLLM, the two major LLM serving engines, have attempted to enforce deterministic inference but face fundamental challenges due to floating-point non-associativity in GEMM kernels, which cau…
A bug is a violation of a specification, and no specification exists that prefix caching's variable logits violate, according to an analysis of vLLM and SGLang issues. The vLLM PR #34046 adds an opt-i…
A developer's optimization campaign on an RTX 3080 Ti cut a GPT-2-small-class transformer's forward pass from 78.2ms to 1.60ms, a 49× speedup, beating torch.compile's 1.72ms and reaching 136,000 token…
TileLang, a high-level Python domain-specific language for designing GPU kernels through TVM, enables developers to implement tensor-core GEMM, fused softmax, FlashAttention, and autotuning while mana…
Torch.compile is not a traditional compiler but a system that intercepts Python bytecode at runtime, extracts compilable regions through a multi-stage pipeline, and stitches them back with eager Pytho…
TornadoVM now natively integrates NVIDIA CUDA libraries cuBLAS, cuFFT, and cuDNN, allowing Java applications to call GPU-accelerated linear algebra and deep learning primitives directly from JIT-compi…
TornadoVM, a Java heterogeneous programming framework, now supports CUDA Graphs, multiple streams, and vendor libraries like cuBLAS, cuFFT, and cuDNN, enabling Java applications to leverage NVIDIA's t…
A developer built Phobos, a tiny scale-free kernel language that compiles to PTX and runs on NVIDIA GPUs, achieving 76% of cuBLAS SGEMM performance on a 2080 SUPER. The language supports tile-DAG desi…
Developer Vincenzo released NanoEuler, a from-scratch GPT-2-scale language model training stack written in C and CUDA without PyTorch or autograd, on June 28, 2026. The 116-million-parameter project i…
A developer released NanoEuler, a GPT-2-scale language model built entirely from scratch in C/CUDA without any machine learning libraries. The project includes a hand-written byte-level BPE tokenizer,…
A developer open-sourced AutoMegakernel, a tool that compiles any HuggingFace model into a single persistent megakernel, reducing overhead by launching one kernel per forward pass. It includes a stati…
Researchers introduced cuTile Rust, a tile-based system for safe, idiomatic GPU kernel authoring in Rust that extends Rust's ownership discipline to GPU kernels. On the NVIDIA B200 GPU, cuTile Rust ac…
NVIDIA's CUTLASS library, a header-only C++ template framework for writing custom CUDA kernels, powers much of the AI infrastructure behind FlashAttention, vLLM, and PyTorch's internal kernels. The li…