A collaboration between AWS, NVIDIA, and Heidi demonstrated a 75% reduction in automatic speech recognition (ASR) inference cost by using NVIDIA CUDA MPS on Amazon EC2 g6e.4xlarge and g7e.4xlarge inst…
AWS, NVIDIA, and Heidi Health report that using NVIDIA CUDA Multi-Process Service (MPS) with NVIDIA Triton Inference Server on Amazon EC2 GPU instances cuts ASR inference GPU infrastructure requiremen…
Netflix's AI Platform team published a detailed account of its LLM serving platform, revealing that version pinning between NVIDIA Triton Inference Server and vLLM, a Python GIL bottleneck, and KV-cac…
Netflix's AI Platform team built an in-house LLM serving stack, running the full pipeline from model deployment through inference inside its existing production environment. The team selected vLLM as …