{"slug": "vllm-sessions-at-pytorch-conference-north-america-2026", "title": "vLLM Sessions at PyTorch Conference North America 2026", "summary": "PyTorch Conference North America 2026, held October 20–21 in San Jose, CA, will feature vLLM across multiple sessions on KV cache management, disaggregated serving, hardware portability, kernel optimization, and PyTorch integration. Notable talks include a session by Mistral AI, Amazon, and Red Hat on KV Push for disaggregated serving, which reportedly reduces time to first token and Pareto-dominates co-located serving on Nemotron, and a session by IBM on native tiered KV cache offloading.", "body_md": "### Featured projects\n\n**TL;DR**\n\nPyTorch Conference North America 2026 features vLLM across sessions on KV cache management and disaggregated serving, hardware portability, kernel optimization, PyTorch integration, Mixture-of-Experts inference, attention, and production serving.\n\n**vLLM at #PyTorchCon NA**\n\nPyTorch Conference North America 2026 comes to San Jose, CA, October 20–21, with technical talks, live demos, lightning talks, sponsored sessions, a keynote, and a Birds of a Feather discussion featuring vLLM.\n\nAcross the program, vLLM appears in sessions on serving architecture and KV cache work, hardware portability, kernel and performance optimization, and PyTorch integration. Additional sessions cover Mixture-of-Experts inference, attention, production deployment, broader application stacks, and open source contribution.\n\n[View the full conference schedule](https://hubs.ly/Q04tDx8f0)\n\n[Register for PyTorch Conference North America 2026](https://hubs.ly/Q04tDw_W0)\n\n**Serving Architecture, KV Cache, and Production Inference**\n\n**A Developer’s Guide to Attention in vLLM**\n\n**Lucas Wilkinson, Red Hat; Matthew Bonanni, Red Hat**\n\n**October 20, 11:45 a.m.–12:10 p.m. | LL20AB | Breakout Session**\n\nThis session explains how vLLM represents, serves, and optimizes attention as models adopt approaches including sliding windows, sparsity, compression, linear variants, and hybrid attention.\n\nThe speakers cover attention backends, KV-cache connectors, and the hybrid memory allocator, along with a recent overhaul of vLLM’s attention abstractions, what changed, and how the new design makes emerging architectures easier and cleaner to support.\n\n**State-of-the-Art KV Transfer for Disaggregated LLM Serving in vLLM**\n\n**Nicolò Lucchesi, Mistral AI; Sunita Nadampalli, Amazon; Zhanqiu Hu, Red Hat**\n\n**October 20, 2:15–2:40 p.m. | LL20CD | Breakout Session**\n\nThis session covers developments in vLLM’s disaggregated serving stack for transferring KV cache between prefill and decode. Topics include hybrid-model transfer with heterogeneous tensor parallelism, bidirectional KV transfer, the KV Push connector, and KV cache leases for reliability.\n\nThe speakers report that KV Push reduces time to first token and that, on Nemotron, disaggregated prefill/decode Pareto-dominates co-located serving across concurrency levels.\n\n**Native Tiered KV Cache Offloading in vLLM: From Storage Offloading to Disaggregated Serving**\n\n**Or Ozeri, IBM**\n\n**October 20, 2:50–3:15 p.m. | LL20CD | Breakout Session**\n\nThis talk presents vLLM’s native tiered KV cache offloading framework, newly integrated upstream with no external dependencies.\n\nThe framework routes transfers through CPU memory as a universal transport hub. The design minimizes GPU transfer overhead, consolidates I/O through a CPU buffer, avoids specialized transfer APIs, and remains independent of KV cache memory layout across hardware, attention backends, parallelism schemes, and model architectures.\n\n**LMCache: a cluster-wide open source solution for LLM prompt caching**\n\n**Kuntai Du, Tensormesh, Inc.**\n\n**October 20, 3:40–3:50 p.m. | LL20CD | Lightning Talk**\n\nLMCache provides prompt caching across inference engines including vLLM, SGLang, and TensorRT-LLM and storage systems including Mooncake, Redis, and AWS S3.\n\nThe session includes a tutorial on deploying LMCache in Kubernetes along with the techniques and research behind its prompt-caching approach.\n\n**vLLM KV Cache Management for Model-Specific Requirements**\n\n**Mengqing Cao, Huawei**\n\n**October 20, 5:30–5:40 p.m. | LL20CD | Lightning Talk**\n\nThis session addresses KV cache requirements that vary across model architectures including MLA, SWA, Eagle, and DeepSeek-V4.\n\nIt proposes a model-customized KV Cache Planner built around a default planner plus model-specific planners for requirements such as spec grouping, block-size derivation, cache tensor creation, and max_model_len adjustment.\n\n**Elastic Expert Parallelism in vLLM**\n\n**Itay Alroy, NVIDIA**\n\n**October 21, 2:50–3:15 p.m. | LL20CD | Breakout Session**\n\nElastic Expert Parallelism enables vLLM deployments to add or remove workers at runtime and redistribute experts across the updated worker set with minimal interruption to serving.\n\nThe session covers communication reconfiguration, CUDA Graph recapture, expert rebalancing through the EP Load Balancer, weight transfer to new GPUs, and coordination with model forward execution. It also covers how NIXL EP enables grow and shrink operations under live traffic, fault detection, reporting, and recovery.\n\n**Prefix Caching for Autoregressive Stages in Multi-Stage Pipelines**\n\n**Ricardo Noriega, Red Hat; Alex Brooks, Red Hat**\n\n**October 21, 4:20–4:45 p.m. | LL20AB | Breakout Session**\n\nThis talk explores Automatic Prefix Caching for Stage Outputs in vLLM-Omni, an approach for extending vLLM’s prefix caching to multi-stage models while minimizing GPU memory cost.\n\nThe approach aligns external CPU tensor caches with vLLM’s native block management. The speakers also describe how vLLM-Omni dynamically discovers cacheable tensors without requiring manual configuration.\n\n**Hardware Portability and Accelerator Backends**\n\n**Sponsored: Unifying Open Source LLM Serving on Google Cloud TPUs with TorchTPU**\n\n**Rob Mulla, Google**\n\n**October 20, 10:40–10:50 a.m. | Community Expo | Demo Theater**\n\nThis 10-minute demo shows TorchTPU’s native, high-performance compilation path for PyTorch models on Cloud TPUs.\n\nThe demo highlights TorchTPU as a unified backend for inference engines including vLLM and SGLang, enabling model deployment through those serving engines with minimal code modifications.\n\n**Sponsored: PyTorch Ecosystem Running Natively on Trainium**\n\n**Maen Suleiman, Amazon Web Services**\n\n**October 20, 10:55–11:05 a.m. | Community Expo | Demo Theater**\n\nThis live demo covers PyTorch workflows on Trainium through TorchNeuron, including training with TorchTitan or Hugging Face Transformers v5, serving with vLLM-Neuron, profiling with Neuron Explorer, and adding NKI kernels directly to PyTorch code.\n\nThe session also demonstrates Neuron Agentic Development, AI-assisted tooling for kernel authoring and optimization.\n\n**One Model Definition, Many Accelerators: Scaling vLLM Across Hardware Without Forks**\n\n**Thomas Parnell, IBM; Richard Zou, Meta**\n\n**October 20, 4:20–4:45 p.m. | LL20CD | Breakout Session**\n\nThis talk presents hardware-agnostic model definitions for vLLM, an approach that separates model logic from hardware execution paths so the same model definition can run across accelerators without forks or per-platform maintenance.\n\nThe design relies on compatibility with torch.compile, well-defined extensibility hooks, and isolation from hardware-specific paths. The speakers show how the approach supports Intel Gaudi/HPU and IBM Spyre without hardware-specific modeling code.\n\n**Portable PyTorch Across AI Accelerators: A Triton Operator Stack from Eager Mode to vLLM**\n\n**Yonghua Lin, Beijing Academy of Artificial Intelligence**\n\n**October 20, 4:55–5:20 p.m. | LL20CD | Breakout Session**\n\nThis session presents FlagOS, an open source system stack using a Triton-based operator, compiler, and runtime layer for PyTorch.\n\nFlagGems implements PyTorch eager-mode operators and LLM-critical kernels in Triton, while the same operator layer connects to vLLM through the vllm-plugin-fl multi-backend plugin.\n\nThe speakers report testing FlagOS on 20+ AI chips and architectures and say it has enabled Day-0 adaptation of models including Qwen3.5, MiniMax-M3, MiniCPM-5, and DeepSeek-V4. They report 5–40% inference performance improvement over original vendor adaptation.\n\n**Efficient MoE LLM Inference on Arm with vLLM and OpenVINO**\n\n**Abhishek Jain, Fujitsu Research of India; N Maajid Khan, Fujitsu Research of India**\n\n**October 20, 4:55–5:05 p.m. | LL21ABC | Lightning Talk**\n\nThis talk presents a vLLM and OpenVINO inference stack optimized for Arm CPUs, including SVE-optimized SDPA and Paged Attention, U8 KV-cache quantization, operator fusion, KleidiAI integration, and optimized threading for 8-bit and 4-bit inference.\n\nFor Mixture-of-Experts models, the speakers introduce a NUMA-aware GatherMatMul operator that combines dynamic token and expert selection with matrix multiplication. Benchmarks on AWS Graviton3e show approximately 2x throughput on GPTOSS/Llama models.\n\n**Integrating the IBM Spyre Accelerator**\n\n**David Grove, IBM; Antoni Viros i Martin, IBM Research; Avery Blanchard, IBM Research**\n\n**October 20, 4:55–5:20 p.m. | LL21DEF | Breakout Session**\n\nTorch-Spyre is an open source project that provides a PyTorch PrivateUse1 device with OpenReg, including an Inductor backend, for the IBM Spyre Accelerator.\n\nThe speakers report that the IBM Spyre Accelerator can now run thousands of models from Hugging Face and vLLM through its PyTorch integration. The session covers the state of Torch-Spyre, functional enablement and performance improvements made in 2026, contributions back to PyTorch, device-specific tensor layouts, and scratchpad-optimized tiling.\n\n**Keynote: Workload Fungibility in the Age of Agents**\n\n**Bill Jia, Google Cloud**\n\n**October 21, 9:15–9:25 a.m. | Grand Ballroom | Keynote**\n\nThis keynote uses TorchTPU to show PyTorch workflows across model development, training, and serving, including serving through vLLM and SGLang.\n\nThe session also demonstrates agentic workflows for moving model workloads from GPUs to TPUs and explores their use for performance optimization tasks including quantization, custom kernel generation, and sharding strategies.\n\n**PyTorch-Native LLM Serving on TPU: SGLang and vLLM**\n\n**Colin Taylor, Meta; Qi Zhou, Google; Angela Yi, Meta**\n\n**October 21, 2:15–2:40 p.m. | LL20CD | Breakout Session**\n\nThis session presents SGLang and vLLM running on TPUs through a new PyTorch-native TPU backend while preserving the serving engines’ schedulers, batching systems, OpenAI-compatible APIs, and torch.compile workflows.\n\nThe speakers cover torch.compile lowering to TPU, Pallas attention, tensor and expert parallelism, Mixture-of-Experts execution, FP8 for large MoE models including Qwen3-Coder-480B, multimodal encoders, prefill/decode disaggregation, and speculative decoding. The speakers say both platforms will be open sourced as of the talk.\n\n### Kernel and Performance Optimization\n\n**High-Velocity GPU Kernel Authoring with CUTLASS Python**\n\n**Michael Goldfarb, NVIDIA; Guray Ozen, NVIDIA**\n\n**October 20, 12:20–12:45 p.m. | 210BF | Breakout Session**\n\nThis talk presents new Python-first capabilities for CUTLASS CuTe DSL, which the speakers report has delivered high-performance GPU kernels in projects including FlashAttention 4, TRT-LLM, vLLM, and FlashInfer.\n\nThe session introduces CuTe DSL extensions, CUTLASS Python Primitives for direct access to low-level hardware instructions, and the Resource and Task Scheduler, a zero-cost metaprogramming framework for static verification of asynchronous primitives.\n\n**Faster LLM Serving Startup with fastsafetensors**\n\n**Takeshi Yoshimura, IBM**\n\n**October 20, 3:25–3:35 p.m. | LL20CD | Lightning Talk**\n\nThis talk presents fastsafetensors, an open source library for accelerating safetensors checkpoint loading in PyTorch inference systems including vLLM.\n\nfastsafetensors removes per-tensor copies, coalesces fragmented I/O, and skips host staging. The speakers report 4.8x to 7.5x faster model loading and up to 28 GB/s of NVMe read throughput.\n\nThe session also covers contributions including parallel loading, 3FS integration, ROCm support, a universal wheel with runtime CUDA/ROCm detection, Windows DirectStorage exploration, and unified-memory support.\n\n**Sponsored: Quantization Showdown: PyTorch Inference Optimization**\n\n**Markell Rawls, Red Hat**\n\n**October 20, 4:10–4:20 p.m. | Community Expo | Demo Theater**\n\nThis live demo examines quantization and speculative decoding using tools including LLM Compressor and vLLM.\n\nThe session stress-tests the techniques under load to examine the performance, cost, and quality tradeoffs involved in production deployments.\n\n**From Weeks to Overnight: Autonomous Day-0 Kernel Bring-Up with Agent Pipelines**\n\n**Xiaogang Gu, Intel; Qun Yang, Intel**\n\n**October 20, 5:30–5:40 p.m. | 210BF | Lightning Talk**\n\nThis session presents a pipeline-driven autonomous system for GPU kernel optimization, with specialized agents handling profiling, analysis, code generation, verification, fixes, benchmarking, and evaluation in isolated contexts.\n\nUsing real vLLM inference workloads, the speakers report reducing kernel bring-up and optimization cycles from weeks to overnight unattended runs.\n\n**Making vLLM Faster on Intel GPUs with Triton Kernels**\n\n**Whitney Tsang, Intel; Artur Fierka, Intel**\n\n**October 21, 12:20–12:30 p.m. | 210BF | Lightning Talk**\n\nThis talk examines Triton kernels for vLLM on Intel GPUs and where Triton can outperform SYCL on inference workloads.\n\nThe speakers focus on unified attention, fused MoE, batched MoE, autotuning, tensor-descriptor-oriented kernel structure, and fusion opportunities. They also discuss where these choices improve end-to-end vLLM throughput and latency and where SYCL remains competitive.\n\n**HiFloat: Democratizing Ultra-Low Precision Training and Inference in PyTorch Ecosystems**\n\n**Yun Zhao, Huawei; Haonan Zhang, Huawei**\n\n**October 21, 2:50–3:00 p.m. | LL20AB | Lightning Talk**\n\nThis session introduces HiFloat8 and HiFloat4 and demonstrates HiFloat-accelerated LLM workflows within DeepSpeed and vLLM.\n\nThe implementation uses PyTorch custom ops, Triton-based kernels, and torch.compile. In the benchmarks presented, the speakers report that HiF8 achieves final-loss parity with FP16 while delivering a 1.5x–1.7x GEMM speedup.\n\n**Portable Paged Attention: From Triton to Helion**\n\n**Burkhard Ringlein, IBM Research**\n\n**October 21, 3:25–3:50 p.m. | 210BF | Breakout Session**\n\nThis session compares Triton and Helion implementations of Paged Attention and presents an experimental Helion attention backend for vLLM.\n\nThe speaker covers differences between the two implementations, algorithm changes needed to achieve matching performance, and optimizations enabled by Helion. Early results indicate that the experimental Helion backend can reduce latency by up to 50% and improve end-to-end throughput versus Triton by up to 10%.\n\n### PyTorch Integration and Compatibility\n\n**Shipping PyTorch and Its Ecosystem: A Modern Release Story**\n\n**Andrey Talman, Meta**\n\n**October 20, 11:45–11:55 a.m. | LL21DEF | Lightning Talk**\n\nThis talk covers changes to PyTorch release engineering, including continuous validation of Triton and vLLM against PyTorch nightlies so ecosystem breakage can surface upstream before the release branch is cut.\n\nThe session also covers a faster, more predictable release process and the use of AI agents to triage CI, separate noise from regressions, and draft fixes, while leaving final decisions with people.\n\n**Sponsored: Making Enterprise Agentic Inference Production-Ready with PyTorch and vLLM**\n\n**Joseph Groenenboom, Red Hat; Tyler Michael Smith, Red Hat**\n\n**October 20, 3:25–3:50 p.m. | LL21ABC | Sponsored Session**\n\nThis session examines reliability, observability, KV cache management, and concurrency requirements for enterprise inference systems.\n\nThe speakers cover a sample of upstream work across PyTorch, vLLM, other Foundation projects, and the broader ecosystem, from core PyTorch build infrastructure to model-serving improvements for tool calling and long-context, multi-turn chat.\n\n**Clearing the Path Towards an ABI Stable PyTorch C++ Extension Ecosystem**\n\n**Sean McGovern, Red Hat; Chris Leonard, Red Hat; Jane Xu, Meta**\n\n**October 21, 2:15–2:40 p.m. | LL21ABC | Breakout Session**\n\nPyTorch’s stable ABI provides a binary-compatible C interface that extensions can target across PyTorch versions without recompilation.\n\nThis session presents tools for identifying and inventorying unstable API usage and applying source-to-source conversion with LLM-assisted follow-up. The speakers demonstrate the process on libraries including vLLM and SGLang.\n\n**From Backed to Unbacked: Sound, Predictable, and Controllable Dynamic Shapes in PyTorch**\n\n**Laith Sakka, Meta**\n\n**October 21, 4:20–4:45 p.m. | LL21ABC | Breakout Session**\n\nThis talk covers unbacked dynamic shapes for explicit graph-capture workflows including vLLM, export, and pre-compilation, as well as JIT deployments where dynamic-shape recompilation is not acceptable.\n\nThe session covers data-dependent errors and branching, work to close the performance gap with backed shapes across TorchBench and vLLM, and APIs for shape constraints and dispatch across compiled artifacts.\n\n**vLLM in Broader Applications and Infrastructure**\n\n**Understanding Modern Vision Language Models**\n\n**Aastha Jhunjhunwala, NVIDIA; Mark Moyou, NVIDIA**\n\n**October 20, 12:20–12:45 p.m. | LL20AB | Breakout Session**\n\nThis session deconstructs five open source vision-language model architectures, covering image tokenization, vision-language fusion, differences between training and inference, fine-tuning, and multi-GPU training.\n\nFor production serving, the speakers cover image-token growth, KV-cache pressure, throughput, and where tools including vLLM fit.\n\n**Sponsored: Hardware-Aware AI: Building Agentic Systems from Cloud to Edge with PyTorch, ExecuTorch**\n\n**Kavya Sri Chennoju, Arm**\n\n**October 20, 12:20–12:45 p.m. | LL20CD | Sponsored Session**\n\nThis session presents a cloud-to-edge workflow combining PyTorch for model development, ExecuTorch for on-device inference, vLLM for scalable LLM serving, and Arm Device Connect for interaction with heterogeneous hardware.\n\nThe live workflow shows foundation models reasoning about tasks, invoking edge models, retrieving live sensor data, and coordinating physical devices.\n\n**Keeping GPUs Busy: High-Speed Storage for PyTorch via fsspec**\n\n**Ankita Luthra, Google; Trinadh Kotturu, Google**\n\n**October 20, 3:40–3:50 p.m. | LL21DEF | Lightning Talk**\n\nThis talk presents Rapid Storage, which brings Google’s Colossus stateful protocol to PyTorch through fsspec and uses persistent gRPC streams to the storage layer.\n\nThe speakers report less than 1 ms random read/write latency, 20x faster data access, 6 TB/s of aggregate throughput, and 10x lower tail latency for random I/O. The integration extends through gcsfs and the broader fsspec ecosystem, including vLLM alongside other data and AI frameworks.\n\n**Sponsored: From Prompt to Physical Action: A Live Hardware-Aware AI Demo with PyTorch, ExecuTorch**\n\n**Kavya Sri Chennoju, Arm**\n\n**October 20, 3:55–4:05 p.m. | Community Expo | Demo Theater**\n\nThis live demonstration combines PyTorch, ExecuTorch, vLLM, and Arm Device Connect in a workflow spanning cloud, edge, and embedded devices.\n\nStarting with a natural-language request, a large language model reasons about the task, discovers available devices, invokes edge AI models, retrieves live sensor data, and coordinates hardware through a unified programming model.\n\n**From PyTorch to Production: Serving a Physics-Constrained Generative Model with ONNX, Ray, and vLLM**\n\n**Arun Sharma, University of Minnesota**\n\n**October 21, 2:50–3:15 p.m. | 210AE | Breakout Session**\n\nThis session follows a physics-constrained generative downscaling model from PyTorch training into a served stack.\n\nThe talk covers model export through torch.onnx and AOTInductor, rectified-flow sampling, physics constraints at inference, and a Ray Train recipe for scaling. The serving path combines ONNX Runtime in Rust, Temporal in Go, and a vLLM agent that calls the downscaler as a tool.\n\n**vLLM Community and Contribution**\n\n**Contributing to Inference OSS That Won’t Stand Still: A BoF on vLLM, llm-d, and the Moving Target**\n\n**Maroon Ayoub, Red Hat; Nili Guy, IBM**\n\n**October 21, 10:35–11:05 a.m. | Community Expo | Birds of a Feather**\n\nThis Birds of a Feather session focuses on contributing to fast-moving inference projects including vLLM and llm-d.\n\nContributors, maintainers, and prospective contributors will compare approaches to landing a first pull request, onboarding contributors, following technical decisions across project channels, and participating in cross-company open source development.\n\n**Explore the Full Program**\n\nThese sessions include both talks centered directly on vLLM and broader sessions where vLLM is part of the serving stack, implementation, hardware integration, optimization work, or application workflow.", "url": "https://wpnews.pro/news/vllm-sessions-at-pytorch-conference-north-america-2026", "canonical_source": "https://pytorch.org/blog/vllm-sessions-at-pytorch-conference-north-america-2026/", "published_at": "2026-08-28 21:30:18+00:00", "updated_at": "2026-08-28 21:49:27.726518+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-research"], "entities": ["PyTorch Conference North America 2026", "vLLM", "Red Hat", "Mistral AI", "Amazon", "IBM", "Huawei", "NVIDIA"], "alternates": {"html": "https://wpnews.pro/news/vllm-sessions-at-pytorch-conference-north-america-2026", "markdown": "https://wpnews.pro/news/vllm-sessions-at-pytorch-conference-north-america-2026.md", "text": "https://wpnews.pro/news/vllm-sessions-at-pytorch-conference-north-america-2026.txt", "jsonld": "https://wpnews.pro/news/vllm-sessions-at-pytorch-conference-north-america-2026.jsonld"}}