At Hot Chips 2026, OpenAI presented Jalapeño, an inference ASIC built with Broadcom, which integrates heterogeneous compute, memory, and network resources on a single chip to handle the three phases o…
SGLang and vLLM, the two major LLM serving engines, have attempted to enforce deterministic inference but face fundamental challenges due to floating-point non-associativity in GEMM kernels, which cau…
A bug is a violation of a specification, and no specification exists that prefix caching's variable logits violate, according to an analysis of vLLM and SGLang issues. The vLLM PR #34046 adds an opt-i…
A vLLM RFC from the llm-d team warns that disaggregated inference deployments, where prefill and decode run on separate schedulers, can trigger recomputation-based preemption inside the decode instanc…
The KV cache has no standard ABI, with vLLM's FlashAttention backend alone reporting its cache shape as a four-dimensional tensor that varies by backend, attention variant, and model family, complicat…
AWS is pairing its Trainium chips with Cerebras CS-3 systems to split transformer inference into prefill and decode phases, with Trainium handling prefill and Cerebras handling decode, shipping as a p…
Anthropic has begun watermarking text output from Claude, embedding imperceptible watermarks in text and attaching C2PA metadata to files, with models launched on or after 2 August 2026 supporting mar…
JAX's Pallas kernel system, which lowers through Triton on GPU and Mosaic on TPU, lets developers write custom kernels when XLA's automatic fusion is insufficient for operations like flash attention, …
JAX, a numerical computing library, is fundamentally a tracing machine that converts pure Python functions into a typed intermediate representation called a jaxpr, refusing to execute code with contro…
Google's open-source TPU compiler, shipped as the Mosaic TPU dialect inside JAX, documents eight explicit memory spaces that programmers must manage, including vector memory (VMEM), scalar memory (SME…
XLA, the compiler behind JAX, TensorFlow, and PyTorch/XLA, optimizes array programs by freezing shapes, statically allocating buffers, and fusing operations against a global cost model, which makes it…
XLA, the compiler under JAX, TensorFlow, and PyTorch/XLA, uses two intermediate representations: classic HLO (a hand-built C++ IR) and MLIR dialects such as StableHLO and CHLO, with a translation laye…
Torch.compile is not a traditional compiler but a system that intercepts Python bytecode at runtime, extracts compilable regions through a multi-stage pipeline, and stitches them back with eager Pytho…
Triton is a compiler with a Python frontend that parses a function's AST, runs it through an MLIR pipeline, and emits a GPU binary, never executing the Python function as Python. The compiler handles …
A cubin (CUDA binary) is a standard ELF64 file with NVIDIA-specific sections that encode everything needed to load and launch a kernel, including machine code, parameter layout, and register allocatio…
A survey of code repository hosting in 2026 finds that GitHub remains the default for most developers, but concerns over DMCA compliance, AI training on code, account suspension, and Microsoft ownersh…
PyTorch, JAX, and compiler-level tools like Enzyme use three different representations for automatic differentiation—runtime tape, functional trace-and-transform, and source (IR) transformation—each w…
MLIR, a compiler infrastructure framework, has become the foundation for numerous machine learning compilers including XLA, Triton, Mojo, Torch-MLIR, IREE, and ONNX-MLIR. It provides a reusable IR con…
VLLM, a large-model inference serving framework, uses Python for control flow but pushes arithmetic into compiled C++ and CUDA kernels to avoid interpreter overhead. The Python/C++ boundary crossing i…
VLIW architecture is experiencing a resurgence driven by machine learning workloads, which rely on regular, statically analyzable computation that VLIW exploits efficiently. Unlike out-of-order design…