AMD and the veRL project released a production-ready reinforcement learning post-training container for AMD Instinct GPUs, supporting MI300 and MI355 series on ROCm, with AITER-accelerated vLLM and SG…
AMD's experiments with speculative decoding in vLLM on AMD Instinct MI300X and MI355X GPUs using the ROCm platform show that output-token throughput gains vary by drafting method, proposal length, mod…
A new guide from Vetted Consumer identifies six bottlenecks that slow local large language models, ranked by impact, with the top cause being the model not fitting in fast memory, which can drop speed…
A developer has ported ROCm 10.0 to AMD's gfx803 architecture (Polaris GPUs), enabling vLLM and llama.cpp to run on older cards like the RX 580. The port, which builds on earlier work with ROCm 6.4.4 …
An enthusiast experiment combined a Strix Halo APU (Ryzen AI MAX+ 395 with 128 GB unified memory and Radeon 8060S iGPU) with two Radeon R9700 32 GB discrete GPUs via an external PCIe switch, achieving…
A Level1Techs forum user reports that running two AMD Radeon RX 9700 Pro GPUs on Windows with llama.cpp and Qwen 3.6 27B Q4_K_M can double token generation speed for complex prompts, reaching 37.8 tok…
Salvatore Sanfilippo, the creator of Redis known as antirez, demonstrated DeepSeek V4 Flash with vision running locally on a Mac M5 Max on September 1, 2026, and announced that Metal, CUDA, and ROCm b…
A hardware-sizing analysis by an unnamed engineer claims that two AMD Radeon PRO R9700 GPUs ($1,300 each) and a Mac Studio M3 Ultra (256GB, $6k) could outperform NVIDIA's DGX Spark (128GB, 273GB/s) fo…
AMD's ATOM and vLLM-ATOM inference engines were optimized for high-interactivity LLM inference, targeting responsiveness for single users rather than batch throughput. The optimizations focus on remov…
A forum user comparing CUDA and ROCm for local AI inference reports that ROCm issues have diminished and is considering an all-AMD build with dual R9700 GPUs by 2027, citing AMD's lower cost. Another …
A developer detailed how to run AI models locally on an AMD Radeon RX 9060 XT using Windows Subsystem for Linux (WSL) and ROCm. The setup involves installing Ollama with its ROCm backend, configuring …
A user with an AMD RX 7900 XT is considering adding lower-end Nvidia RTX GPUs to run multiple local AI models on different cards, but is concerned about potential issues with running ROCm and CUDA sid…
A developer reports that running Qwen 3.8 27B at Q5_K_M quantization on a dual AMD Radeon RX 7900 XT and 7800 XT setup achieves 20 tokens per second with a 256k context window, enabling autonomous mul…
AMD detailed the Instinct MI400 series GPU architecture at Hot Chips 2026, highlighting the MI455X silicon with 432 GB of HBM4 memory, 23.3 TB/s bandwidth, and a peak MXFP4 performance of 40.26 petafl…
A new setup guide from Minisforum details running ROCmFP4-quantized models with Multi-Token Prediction (MTP) on the N5 Max AI NAS, powered by the AMD Ryzen AI MAX+ 395 (16C/32T) with Radeon 8060S (gfx…
A bug is a violation of a specification, and no specification exists that prefix caching's variable logits violate, according to an analysis of vLLM and SGLang issues. The vLLM PR #34046 adds an opt-i…
A user reports running large language models on a 128GB Bosgame M5 Strix Halo mini PC purchased used for 1800€ on eBay, achieving 30 tokens per second decode with Qwen 3.8 27B and 50 tokens per second…
A Proxmox VE 9 host with two Radeon AI PRO R9700 32 GB GPUs achieved working peer-to-peer (P2P) transfers at ~13 GiB/s and ran Qwen3.8 27B UD-Q8_K_XL at ~31 t/s (non-MTP) and ~49-50 t/s (MTP n=3) usin…
A developer's final part of a CUDA/ROCm optimization series details implementing Mixture of Experts (MoE) inference, including a custom top-k gating kernel, expert parallelism with All-to-All communic…
A developer building a home AI server from e-waste GPUs details the process of optimizing multi-GPU performance for running large language models, focusing on llama.cpp settings and existing technique…