PyTorch's torch._assert_async API enables asynchronous assertion on GPU device streams, avoiding graph breaks in torch.compile. The API accepts a boolean tensor and reports failures only when the GPU …
Vector quantization and product quantization offer an alternative to scaling-based neural network quantization, achieving high compression ratios by mapping high-dimensional vectors to a finite codebo…
PyTorch's multiprocessing module provides a CUDA Inter-Process Communication (IPC) API that enables sharing model weights across multiple processes for inference, avoiding duplication in GPU VRAM. The…
Predicated execution and conditional execution are two approaches to handling control flow in programming, with predicated execution running all instructions but committing only those meeting a condit…
TorchRec's KeyedJaggedTensor, designed to efficiently combine sparse features in recommendation systems without padding, introduces GPU-CPU synchronization that degrades system performance. The data t…
PyTorch users can now implement custom operations in C++ and CUDA for use in both Python and C++ inference programs, with automatic device dispatch between CPU and CUDA implementations. The approach s…
PyTorch AOTInductor now compiles exported programs with hybrid CPU-GPU execution plans into a single executable package, eliminating the need to manually split models into separate device sub-models. …
PyTorch has introduced transparent tracing and compilation for Triton kernels, allowing custom operations to be visible to the compiler for optimization. The framework now supports compiling Triton ke…
PyTorch introduced a "fake export" method that allows developers to verify the exportability of large deep learning models using `torch.export` APIs without requiring actual GPU memory. The approach u…