A benchmark of vLLM 0.30.0, SGLang 0.5.20, and TensorRT-LLM 1.2.1 on a single H100 SXM GPU with 8 CPU cores found no host-bound regime across Qwen 3 0.6B-32B models at batch sizes from 1 to 256, with …
NVIDIA published a step-by-step tutorial demonstrating how to debug, benchmark, and optimize CUDA code using modern tools such as Compute Sanitizer, Nsight Systems, NVTX, CUB, and CCCL, applied to an …
Host overhead, caused by the CPU blocking the GPU, is a major source of inefficiency in AI inference, leading to low GPU kernel utilization and doubling GPU costs when at 50%. Modal recommends using t…
NVIDIA published a technical blog post explaining how kernel fusion in CUDA can optimize GPU code by combining multiple operations into a single kernel, reducing memory traffic and launch overhead. Th…
NVIDIA used Nsight Systems and Nsight Compute to optimize its NuRec neural reconstruction pipeline, reducing reconstruction times from hours toward a goal of real-time performance for autonomous vehic…
An eBPF agent that attaches to the CUDA runtime, CUDA driver, and Linux kernel scheduler simultaneously can trace a GPU stall back to the exact Python source line that triggered it. The tool correlate…
NVIDIA released CUDA 13.3, introducing tile programming in C++ that automates low-level GPU management for optimized kernel development across all supported architectures. The update also includes CUD…