Migrating vLLM to the LibTorch Stable ABI

VLLM v0.33.0 now publishes LibTorch ABI Stable CUDA wheels, letting the same vLLM wheel run across compatible PyTorch releases without a rebuild or relink. The migration relied on the stable C shim, C++ wrappers and torch::headeronly utilities, and used the torch_call_dispatcher C shim to call ATen ops such as is_pinned() in an ABI-stable way; PyTorch 2.14 subsequently added a stable C++ is_pinned() method. The vLLM team reports no noticeable runtime regression, though it notes the stable ABI covers only a subset of LibTorch and adds small host-side type-conversion and dispatcher overhead that is negligible under CUDA Graphs.

Migrating vLLM to the LibTorch Stable ABI Introduction vLLM relies heavily on PyTorch, with frequent calls to LibTorch in its C++ backend. As a result, changes to LibTorch’s binary interface may require vLLM to be rebuilt and relinked, even when the APIs remain unchanged. What if we could update LibTorch without rebuilding vLLM? That’s where the LibTorch Stable ABI comes in. With a LibTorch Stable ABI, updating LibTorch does not require rebuilding or updating vLLM in order to keep using it. Instead, we could swap out the torch wheel and import it directly into our vLLM package. As of v0.33.0, vLLM publishes LibTorch ABI Stable CUDA wheels. Note: Compatibility between vLLM and PyTorch still depends on other factors, including Python APIs, GPU-related requirements such as the minimum supported CUDA version , and third-party dependencies. The LibTorch Stable ABI addresses binary compatibility but does not eliminate these other constraints. What Is the LibTorch Stable ABI? An application binary interface ABI defines how compiled code interacts with a library: how functions are called, data is laid out, and symbols are named. Even if an API looks the same in source code, an ABI change can force extensions to rebuild. A stable ABI preserves a supported set of binary interfaces across compatible versions, so extensions that use them can keep working without a rebuild. PyTorch’s LibTorch Stable ABI has three layers: a stable C shim exposing selected LibTorch APIs, C++ wrappers that make the shim easier to use, and headeronly utilities in torch::headeronly that do not link against libtorch.so . Why Stable ABI Matters Before the Stable ABI, a new PyTorch release could require a vLLM rebuild, delaying support for the new version and its features. With the Stable ABI, the same vLLM wheel can work across compatible PyTorch releases, letting users upgrade sooner and reducing rebuilds for package distributors. For developers, this means you can update and rebuild PyTorch, then test those changes in vLLM without rebuilding vLLM. Tradeoffs The LibTorch Stable ABI is a limited subset of all of LibTorch. While many common ops are already supported through the C shim and through the higher level C++ convenience wrappers, not all are. For example, is pinned lacked a stable equivalent during the migration. Thankfully, there is a general-purpose torch call dispatcher C shim that allows us to call ATen ops through the dispatcher in an ABI stable manner This let us maintain behavior and achieve ABI stability at the same time. This migration also helped inform PyTorch which APIs were commonly used, and in version 2.14 they added a stable C++ is pinned method. Before: direct LibTorch method call if cpu tensor.is pinned After: ABI stable dispatcher call std::array<StableIValue, 2 is pinned stack{ torch::stable::detail::from cpu tensor , torch::stable::detail::from std::nullopt }; TORCH ERROR CODE CHECK torch call dispatcher "aten::is pinned", "", is pinned stack.data , TORCH ABI VERSION ; if torch::stable::detail::to<bool is pinned stack 0 The second tradeoff is a small amount of additional host-side overhead. The stable C shim adds type-conversion overhead and, for some wrappers that rely on torch call dispatcher , operator lookup and boxing. This overhead is usually negligible and disappears after warmup under CUDA Graphs, but it can matter in eager mode when repeatedly launching very fast kernels. We saw no noticeable runtime regression during the migration. Migrating vLLM to the Stable ABI We have broken down the migration to the Stable ABI into a series of steps: 1. Assess the situation First, we assessed whether vLLM could migrate using the stable APIs available at the time. We inventoried its PyTorch API calls, identified gaps, and planned the stable APIs needed to fill them. This work is outlined in the original RFC https://github.com/vllm-project/vllm/issues/26946 for updating vLLM’s CUDA build to use the LibTorch Stable ABI. 1. Migrate AutogradFunctions and meta functions to Python As of today, the LibTorch Stable ABI does not support stable SymInt, so meta functions need to be written in Python instead of C++. Luckily, vLLM already had most of its meta functions in Python, so migrating the few that were not was relatively simple and made the project more consistent. Similarly, the Stable ABI does not expose PyTorch’s autograd machinery, so AutogradFunctions also need to be written in Python. This matters less for an inference engine like vLLM, but if a backward function is implemented, it would need to be in Python. 1. Set up necessary scaffolding Since vLLM is such a big project, with over 175 .cpp/.cu source files plus another 120+ headers in its CUDA build alone, migrating everything in one go would have been unwieldy and difficult to review. Instead, we created a separate stable library see CMakeLists.txt updates in the first migration PR 31509 https://github.com/vllm-project/vllm/pull/31509/changes diff-1e7de1ae2d059d21e1dd75d5812d5a34b0222cef273b7c3a2af62eb747f9d20a , and we migrated the kernels to this library in manageable chunks. The TORCH TARGET VERSION build flag helps keep the new library LibTorch ABI stable by causing the build to fail if it includes any unstable LibTorch headers. 1. Make sure you register ops through the dispatcher As of today, registering your LibTorch Stable ops requires registration through STABLE TORCH LIBRARY . This is very familiar if you already register your ops through TORCH LIBRARY , which vLLM does. The biggest difference with the new stable registration is that the implementation needs to be wrapped with TORCH BOX . This generates an adapter that boxes and unboxes the C++ data types into the generic StableIValue types that can cross PyTorch’s Stable ABI. If your kernels are registered through another mechanism, such as pybind11, we recommend first converting the registration to TORCH LIBRARY before registering through STABLE TORCH LIBRARY more on this below in the Challenges Along the Way section . 1. Migrate each source file to use stable APIs Once steps 1–4 are in place, we can migrate each source file to stable APIs. Any PyTorch headers we include should come from the C++ stable wrappers, the headeronly utilities, or the C shim itself: - torch/csrc/stable - torch/headeronly - torch/csrc/stable/c/shim.h or torch/csrc/inductor/aoti torch/c/shim.h The source file changes are mostly mechanical. For example: - torch::Tensor → torch::stable::Tensor - at::kInt → torch::headeronly::ScalarType::Int - t.view {...} → torch::stable::view t, {...} To help migrate the source files, we created skills for AI agents that map existing APIs to their stable equivalents and flag common mistakes, such as changes unrelated to the migration. These skills also support the earlier steps 1–4 . They have been useful in our work, but we have not yet developed evaluations to measure their effectiveness see abi-stable-skills https://github.com/janeyx99/abi-stable-skills . 1. Verify vLLM is used by millions of users across multiple industries, and we do not want to interrupt any of their processes with the migration. Therefore, the most important step in this list is to verify that the migration did not make any significant changes. This means running appropriate unit and performance tests and manually reviewing each migration diff line by line to make sure there were no unintended updates. Along with verifying correctness, we also need to verify that the new library is actually LibTorch ABI Stable. For this, we use torch-abi-audit https://pypi.org/project/torch-abi-audit/ thanks Ralf Gommers . This tool inspects .so files and categorizes stable and unstable PyTorch symbols. If there are no unstable symbols, the library is LibTorch ABI Stable. It also reports the minimum LibTorch version that provides all the required stable symbols for example, min-torch=2.11.0 . We used it to track migration progress and check for regressions, and now use it in vLLM CI to ensure the wheel remains LibTorch ABI Stable. torch-abi-audit vllm Package: vllm Root: /workspaces/vllm container/vllm/vllm Torch ABI: UNSTABLE Min torch: 2.11.0 CPython ABI: mixed Extensions: 10 Bundled libs: 0 -- extensions -- STABLE abi3-ok C stable libtorch.abi3.so stable shim=88, unstable=0 min-torch=2.11.0 UNSTABLE abi3-ok flashmla C.abi3.so stable shim=0, unstable=70 UNSTABLE abi3-ok flashmla extension C.abi3.so stable shim=0, unstable=68 STABLE abi3-ok moe C stable libtorch.abi3.so stable shim=72, unstable=0 min-torch=2.11.0 UNSTABLE abi3-ok qutlass C.abi3.so stable shim=0, unstable=59 NO-TORCH abi3-ok cumem allocator.abi3.so NO-TORCH abi3-ok spinloop.abi3.so UNSTABLE uses-private-api third party/deep gemm/ C.cpython-312-x86 64-linux-gnu.so stable shim=0, unstable=59 UNSTABLE abi3-ok vllm flash attn/ vllm fa2 C.abi3.so stable shim=0, unstable=85 UNSTABLE abi3-ok vllm flash attn/ vllm fa3 C.abi3.so stable shim=0, unstable=81 torch-abi-audit before the dependencies were migrated vLLM Build Dependencies Migrating vLLM source alone does not establish that the entire vLLM wheel is ABI stable. For the wheel to be completely stable, its compiled dependencies must be LibTorch ABI Stable as well. vLLM currently has seven dependencies that also depend on LibTorch. These are: | Dependency | Migrated on vLLM Fork | Migrated Upstream | |---|---|---| | FlashMLA | ✅ | ✅ | | DeepGEMM | ✅ | 🚧 Work in progress | | FlashAttention-2 | ✅ | 🚧 PR open to make it stable | | FlashAttention-3 | ✅ | ✅ | | DeepSelect | ✅ | 🚧 Work in progress | | FlashKDA | ✅ | ❌ | | QuTLASS | N/A | ✅ | After the in-tree vLLM code had been migrated, each of these dependencies was migrated to the LibTorch Stable ABI as well, making the entire vLLM CUDA wheel LibTorch ABI Stable. torch-abi-audit vllm Package: vllm Root: /workspaces/vllm container/vllm/vllm Torch ABI: STABLE Min torch: 2.13.0 CPython ABI: abi3 Extensions: 13 Bundled libs: 0 -- extensions -- STABLE abi3-ok C stable libtorch.abi3.so stable shim=88, unstable=0 min-torch=2.11.0 STABLE abi3-ok deepselect C.abi3.so stable shim=40, unstable=0 min-torch=2.10.0 STABLE abi3-ok flashkda C.abi3.so stable shim=43, unstable=0 min-torch=2.11.0 STABLE abi3-ok flashmla C.abi3.so stable shim=66, unstable=0 min-torch=2.11.0 STABLE abi3-ok flashmla extension C.abi3.so stable shim=63, unstable=0 min-torch=2.11.0 STABLE abi3-ok moe C stable libtorch.abi3.so stable shim=72, unstable=0 min-torch=2.11.0 STABLE abi3-ok qutlass C.abi3.so stable shim=63, unstable=0 min-torch=2.13.0 NO-TORCH abi3-ok cumem allocator.abi3.so NO-TORCH abi3-ok fs io C.abi3.so NO-TORCH abi3-ok spinloop.abi3.so STABLE abi3-ok third party/deep gemm/ C extension.abi3.so stable shim=85, unstable=0 min-torch=2.11.0 STABLE abi3-ok vllm flash attn/ vllm fa2 C.abi3.so stable shim=69, unstable=0 min-torch=2.10.0 STABLE abi3-ok vllm flash attn/ vllm fa3 C.abi3.so stable shim=63, unstable=0 min-torch=2.10.0 torch-abi-audit after dependencies were migrated Challenges Along the Way The biggest challenge was keeping pace with a large, rapidly changing project. Upstream files often changed while our migration patches were in progress, so we had to update those patches before they could merge. vLLM’s external dependencies also meant coordinating across projects and building support for the migration. DeepGEMM: A Migration Case Study DeepGEMM brought several of these challenges together. While our migration was in progress, DeepGEMM moved from version 1 to version 2 and added DeepJIT, another LibTorch-dependent project that also needed to be migrated. vLLM then switched from the upstream deepseek-ai/DeepGEMM repository to its own vllm-project/DeepGEMM fork, requiring us to redo the migration on the new fork. Those upstream changes were only part of the challenge. Because DeepGEMM registered its operators through pybind11, we also needed to move its registration to TORCH LIBRARY before adopting the Stable ABI. For example, the original binding of fp8 fp4 paged mqa logits was: m.def "fp8 fp4 paged mqa logits", &fp8 fp4 paged mqa logits, py::arg "q" , ..., py::arg "max context len" , ... ; where q is a std::tuple<Tensor, std::optional<Tensor & and max context len is a const int& . To make these arguments compatible with TORCH LIBRARY , we added a wrapper that accepts the tuple elements separately and uses int64 t in its C++ signature: js static torch::Tensor fp8 fp4 paged mqa logits const torch::Tensor& q, const c10::optional<torch::Tensor & q sf, ... const int64 t& max context len, ... { return attention::fp8 fp4 paged mqa logits std::make tuple q, q sf , ..., static cast<int max context len , ... ; } This can now be registered with TORCH LIBRARY as: TORCH LIBRARY FRAGMENT deep gemm, m { m.def "fp8 fp4 paged mqa logits Tensor q, Tensor? q sf, ..., int max context len, ... - Tensor" ; } TORCH LIBRARY IMPL deep gemm, CUDA, m { m.impl "fp8 fp4 paged mqa logits", TORCH FN fp8 fp4 paged mqa logits ; } The Python interface unpacks the tuple before calling the registered operator: python def fp8 fp4 paged mqa logits q, ..., max context len, ... : q fp, q sf = q 0 , q 1 return torch ops.fp8 fp4 paged mqa logits q fp, q sf, ..., max context len, ... This lets the registration use TORCH LIBRARY without changing the user-facing API. After DeepGEMM was migrated to TORCH LIBRARY, the Stable ABI migration followed a similar process as the other migrations. We replaced its unstable APIs and updated the operator registrations: STABLE TORCH LIBRARY FRAGMENT deep gemm, m { m.def "fp8 fp4 paged mqa logits Tensor q, Tensor? q sf, ..., int max context len, ... - Tensor" ; } STABLE TORCH LIBRARY IMPL deep gemm, CUDA, m { m.impl "fp8 fp4 paged mqa logits", TORCH BOX fp8 fp4 paged mqa logits ; } Because DeepGEMM was vLLM’s final build dependency to be migrated, completing this work made the entire vLLM wheel LibTorch ABI Stable Example Run on Newer PyTorch python stable abi demo.py import os os.environ.setdefault "VLLM USE FLASHINFER SAMPLER", "0" import torch import vllm from vllm import LLM, SamplingParams print f"PyTorch runtime: {torch. version }" print f"vLLM: {vllm. version }" llm = LLM model="HuggingFaceTB/SmolLM2-135M-Instruct", max model len=128, gpu memory utilization=0.1, enforce eager=True, outputs = llm.generate "The capital of France is" , SamplingParams temperature=0, max tokens=8 , print outputs 0 .outputs 0 .text Build vLLM and run the demo with PyTorch 2.14.1: python stable abi demo.py PyTorch runtime: 2.14.1+cu132 vLLM: 0.1.dev24716+gcb95d994f.d20261105 Paris. Paris is a city known for Update the PyTorch wheel to torch==2.15.0 : uv pip install torch==2.15.0 \ --index-url https://download.pytorch.org/whl/cu132 Rerun demo without rebuilding vLLM: python stable abi demo.py PyTorch runtime: 2.15.0+cu132 vLLM: 0.1.dev24716+gcb95d994f.d20261105 Paris. Paris is a city known for Note: Routine vLLM initialization logs, progress bars, and unrelated deprecation warnings have been omitted from the outputs above for clarity. Future Work Now that vLLM’s CUDA build uses the LibTorch Stable ABI, our next steps are to make all of vLLM’s builds ABI stable, including the CPU and ROCm builds. These migrations are smaller than the CUDA migration but remain important to the vLLM ecosystem. The ROCm build shares many source files with the CUDA build: hipify converts .cu files to .hip files, so these files already use the LibTorch Stable ABI. Only a few ROCm-specific files remain to be migrated. We are also expanding the stable API surface so that vLLM and its dependencies no longer need wrappers. As the API surface grows, we will also be updating PyTorch’s upstream tests to ensure the APIs remain LibTorch ABI Stable. Acknowledgement to main contributors - vLLM Community: Shengqi Chen https://github.com/Harry-Chen main coordinator , Jiangyun Zhu https://github.com/ZJY0516 DeepSelect fork migration , Thien Tran https://github.com/gau-nernst FlashKDA fork migration , Roberto L. Castro https://github.com/LopezCastroRoberto upstream QuTLASS migration , Kaichao You https://github.com/youkaichao roadmap , and all maintainers who reviewed our PRs and provided suggestions. - PyTorch Community: Jane Xu https://github.com/janeyx99 project lead, FlashMLA migration in both fork and upstream, abi-stable-skills https://github.com/janeyx99/abi-stable-skills project , Chris Leonard https://github.com/cleonard530 major migration work , Mikayla Gawarecki https://github.com/mikaylagawarecki early-stage migration work for in-tree sources , Ralf Gommers https://github.com/rgommers torch-abi-audit https://pypi.org/project/torch-abi-audit/ project . - DeepSeek: Chenggang Zhao https://github.com/LyricZhao , Yichen Yan https://github.com/oraluben , Shengyu Liu https://github.com/interestingLSY .