Kourosh Hakhamaneshi

Engineers achieved up to 67% cost savings and 2.7x better goodput by using Prefill-Decode disaggregation with Ray and vLLM on AMD MI325X GPUs, separating prefill and decode phases onto dedicated hardw…

Ray Serve LLM and vLLM on AMD MI325X achieve up to 67% cost savings by disaggregating prefill and decode phases in LLM serving, separating them onto dedicated GPUs to eliminate interference and improv…