FlashInfer<sup>1</sup> is an open-source library of attention and large language model (LLM) serving kernels. FlashInfer on ROCm™ brings its API to AMD Instinct™ GPUs, and this release continues from the prefill-attention release. Cascade attention, multi-head latent attention (MLA), and the standalone rotary position embedding (RoPE) ops, all in progress then, have now landed. New in this release are 8-bit floating-point (FP8) prefill, POD-Attention (prefill and decode overlapped in one kernel), fused mixture-of-experts (MoE), and AI Tensor Engine for ROCm (AITER) decode under CUDA graph capture. All of it runs on CDNA3 (gfx942: MI300X and MI325X)<sup>2</sup> and CDNA4 (gfx950: MI350X and MI355X)<sup>3</sup> GPUs.
FlashInfer on ROCm 10.1: The FlashInfer API with Automatic Kernel Routing
FlashInfer on ROCm 10.1 brings the FlashInfer attention and LLM serving kernel API to AMD Instinct GPUs with automatic kernel routing, adding FP8 prefill, POD-Attention (prefill and decode overlapped in one kernel), fused mixture-of-experts (MoE), and AITER decode under CUDA graph capture. The release also lands cascade attention, multi-head latent attention (MLA), and standalone rotary position embedding (RoPE) ops that were in progress in the earlier prefill-attention release. It runs on CDNA3 (gfx942: MI300X and MI325X) and CDNA4 (gfx950: MI350X and MI355X) GPUs.
FlashInfer on ROCm 10.1 adds FP8 prefill, POD-Attention, fused MoE and AITER decode under CUDA graph capture, plus cascade attention, MLA and standalone RoPE, on gfx942 (MI300X/MI325X) and gfx950 (MI350X/MI355X) GPUs.
Run your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.