A forum user argues that 32GB of VRAM is the practical threshold for useful large language model use, noting that 24GB cards work but with limited context size. The user highlights that NVIDIA offers …
A developer's final part of a CUDA/ROCm optimization series details implementing Mixture of Experts (MoE) inference, including a custom top-k gating kernel, expert parallelism with All-to-All communic…
DeepSeek V4 Flash, a 284B-parameter mixture-of-experts model with 256 routed experts and FP4 expert weights, was successfully deployed on eight AMD Radeon AI PRO R9600D GPUs (32 GB each, 256 GB total)…
A developer's third installment in a GPU optimization series details implementing distributed training for large language models using CUDA and ROCm, covering All-Reduce, Ring-AllReduce, and ZeRO shar…
AMD's QuickReduce library now supports INT3 quantization for all-reduce communication in multi-GPU LLM inference, achieving a 22% reduction in on-wire data volume compared to INT4 on AMD Instinct MI35…
AMD has brought PyTorch Monarch to its Instinct GPUs with ROCm, enabling single-controller distributed training for large language models. The port addresses reliability challenges at scale by providi…
AMD Strix Halo cluster setup guide details how to configure a two-node system linked via Intel E810 RoCE v2 for distributed vLLM inference using Tensor Parallelism. The guide covers hardware prerequis…
Maincode, an Australian AI company, built a custom inference engine in Rust for the AMD MI355X GPU that bypasses the ROCm runtime entirely, achieving lower overhead on critical operations like tensor-…