NIXL: NVIDIA Inference Xfer Library

NVIDIA has released the NVIDIA Inference Xfer Library (NIXL), an open-source library designed to accelerate point-to-point communications in AI inference frameworks such as NVIDIA Dynamo, with support for Linux environments including Ubuntu 22.04/24.04 and Fedora. The library, available via PyPI with `pip install nixl`, provides a modular plug-in architecture that abstracts over CPU/GPU memory and file, block, and object storage, and requires a C++20 compiler and UCX version 1.23.x for optimal performance.

NVIDIA Inference Xfer Library NIXL is targeted for accelerating point to point communications in AI inference frameworks such as NVIDIA Dynamo, while providing an abstraction over various types of memory e.g., CPU and GPU and storage e.g., file, block and object store through a modular plug-in architecture. - NIXL overview https://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md - Core concepts/architecture overview docs/nixl.md - Python API https://github.com/ai-dynamo/nixl/blob/main/docs/python api.md - Python API usage and examples docs/python api.md - Backend guide https://github.com/ai-dynamo/nixl/blob/main/docs/BackendGuide.md - Backend/plugin development guide docs/BackendGuide.md - Telemetry https://github.com/ai-dynamo/nixl/blob/main/docs/telemetry.md - Observability and telemetry details docs/telemetry.md - Doxygen guide https://github.com/ai-dynamo/nixl/blob/main/docs/doxygen/nixl doxygen.md - API/class diagrams overview docs/doxygen/nixl doxygen.md - Doxygen images https://github.com/ai-dynamo/nixl/tree/main/docs/doxygen - Diagram assets docs/doxygen/ - NIXLBench docs https://github.com/ai-dynamo/nixl/blob/main/benchmark/nixlbench/README.md - Benchmark usage guide benchmark/nixlbench/README.md - KVBench docs https://github.com/ai-dynamo/nixl/tree/main/benchmark/kvbench/docs - KVBench workflows and tutorials benchmark/kvbench/docs/ NIXL is supported on a Linux environment only. It is tested on Ubuntu 22.04/24.04 and Fedora. macOS and Windows are not currently supported; use a Linux host or container/VM. The nixl python API and libraries, including UCX, are available directly through PyPI. For example, if you have a GPU running on a Linux host, container, or VM, you can do the following install: Install with: pip install nixl This installs both CUDA 12 and CUDA 13 backends. At runtime, the correct backend is selected automatically based on the CUDA version reported by PyTorch. NIXL requires a C++20 compatible compiler GCC = 11 or Clang = 14 . $ sudo apt install build-essential cmake pkg-config $ sudo dnf install gcc-c++ cmake pkg-config $ pip3 install meson ninja pybind11 tomlkit NIXL was tested with UCX version 1.23.x. GDRCopy https://github.com/NVIDIA/gdrcopy is available on Github and is necessary for maximum performance, but UCX and NIXL will work without it. bash $ git clone https://github.com/openucx/ucx.git $ cd ucx $ git checkout v1.23.x $ ./autogen.sh $ ./contrib/configure-release-mt \ --enable-shared \ --disable-static \ --disable-doxygen-doc \ --enable-optimizations \ --without-avx \ --enable-cma \ --enable-devel-headers \ --with-cuda=<cuda install \ --with-verbs \ --with-dm \ --with-gdrcopy=<gdrcopy install $ make -j $ make -j install-strip $ ldconfig NIXL can use ETCD for metadata distribution and coordination between nodes in distributed environments. To use ETCD with NIXL: bash $ sudo apt install etcd etcd-server etcd-client Or use Docker $ docker run -d -p 2379:2379 quay.io/coreos/etcd:v3.5.1 Installed from https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3 https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3 bash $ sudo apt install libgrpc-dev libgrpc++-dev libprotobuf-dev protobuf-compiler-grpc $ sudo apt install libcpprest-dev $ git clone https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3.git $ cd etcd-cpp-apiv3 $ mkdir build && cd build $ cmake .. $ make -j$ nproc && make install Some plugins may have additional build requirements, see them here: bash $ meson setup <name of build dir $ cd <name of build dir $ ninja $ ninja install bash $ meson setup <name of build dir bash $ meson setup <name of build dir --buildtype=debug bash Example with custom options $ meson setup <name of build dir \ -Dbuild docs=true \ Build Doxygen documentation -Ducx path=/path/to/ucx \ Custom UCX installation path -Dinstall headers=true \ Install development headers -Ddisable gds backend=false Enable GDS backend Common build options: - build docs : Build Doxygen documentation default: false - ucx path : Path to UCX installation default: system path - install headers : Install development headers default: true - disable gds backend : Disable GDS backend default: false - cudapath inc , cudapath lib : Custom CUDA paths - static plugins : Comma-separated list of plugins to build statically - enable plugins : Comma-separated list of plugins to build e.g. -Denable plugins=UCX,POSIX . Cannot be used with disable plugins . - disable plugins : Comma-separated list of plugins to exclude e.g. -Ddisable plugins=GDS . Cannot be used with enable plugins . - wheel variant : Override the Python wheel variant suffix e.g. -Dwheel variant=rocm yields nixl rocm . Empty default = autodetect from the CUDA major version. NIXL itself builds vendor-neutrally; CPU-side hardware detection hwInfo::numAmdGpus discovers AMD GPUs via PCI vendor 0x1002 whether or not a ROCm toolchain is present. GPU-side ROCm/HIP build support is available for nixlbench and UCX plugin unit tests. When packaging a ROCm wheel, pass -Dwheel variant=rocm so the wheel is named nixl rocm . Building with ROCm support: bash For UCX unit tests with ROCm $ meson setup build -Drocm path=/opt/rocm Or specify a custom ROCm path $ meson setup build -Drocm path=/custom/path/to/rocm Plugins on ROCm hosts CUDA toolchain absent : - UCX — primary transport for AMD GPU memory requires UCX built with --with-rocm . - POSIX , OBJ , AZURE BLOB , HF3FS , MOONCAKE , GUSLI , UCCL — vendor-neutral; build unchanged. - GDS / GDS MT , GPUNETIO , LIBFABRIC with -DHAVE CUDA — skip automatically because their CUDA / cuFile / DOCA dependencies are not found. Known gaps will be addressed in follow-up PRs : - LIBFABRIC plugin disabled on ROCm pending header refactor. - No NVSHMEM-equivalent backend yet rocSHMEM analog is a candidate for a future plugin . There are a few environment variables that can be set to configure the build: - NIXL NO STUBS FALLBACK : If not set or 0, build NIXL stub library if the library build fails If you have Doxygen installed, you can build the documentation: bash Configure with documentation enabled $ meson setup <name of build dir -Dbuild docs=true $ cd <name of build dir $ ninja Documentation will be generated in <name of build dir /html After installation ninja install , documentation will be available in <prefix /share/doc/nixl/ NIXL provides Python bindings through pybind11. For detailed Python API documentation, see docs/python api.md https://github.com/ai-dynamo/nixl/blob/main/docs/python api.md . The preferred way to install the Python bindings is through pip from PyPI: pip install nixl This installs both CUDA 12 and CUDA 13 backends. At runtime, the correct backend is selected automatically based on the CUDA version reported by PyTorch. Prerequisites: - uv : https://docs.astral.sh/uv/getting-started/installation/ https://docs.astral.sh/uv/getting-started/installation/ - tomlkit : https://pypi.org/project/tomlkit/ https://pypi.org/project/tomlkit/ - PyTorch : https://pytorch.org/get-started/locally/ https://pytorch.org/get-started/locally/ uv is always required even if you have another kind of Python virtual environment manager or if you are using a system-wide Python installation without using a virtual environment. Example with uv Python virtual environment: curl -LsSf https://astral.sh/uv/install.sh | sh export PATH="$HOME/.local/bin:${PATH}" uv venv .venv --python 3.12 source .venv/bin/activate uv pip install tomlkit Example with python-virtualenv: curl -LsSf https://astral.sh/uv/install.sh | sh export PATH="$HOME/.local/bin:${PATH}" python3 -m venv .venv source .venv/bin/activate pip install tomlkit Example with system-wide Python installation without using a virtual environment: curl -LsSf https://astral.sh/uv/install.sh | sh export PATH="$HOME/.local/bin:${PATH}" pip install tomlkit Then install PyTorch following the instructions on the PyTorch website: https://pytorch.org/get-started/locally/ https://pytorch.org/get-started/locally/ After installing the prerequisites, you can build and install the NIXL binaries and the Python bindings from source. You have to: 1. Build NIXL binaries and install them 2. Build and install the CUDA platform-specific package nixl-cu12 or nixl-cu13 3. Build and install the nixl meta-package For CUDA 12: pip install . meson setup build ninja -C build install pip install build/src/bindings/python/nixl-meta/nixl- -py3-none-any.whl For CUDA 13: pip install . ./contrib/tomlutil.py --wheel-name nixl-cu13 pyproject.toml meson setup build ninja -C build install pip install build/src/bindings/python/nixl-meta/nixl- -py3-none-any.whl To check if the installation is successful, you can run the following command: python python3 -c "import nixl; agent = nixl.nixl agent 'agent1' " which should print: 2026-01-08 13:36:27 NIXL INFO api.py:363 Backend UCX was instantiated 2026-01-08 13:36:27 NIXL INFO api.py:253 Initialized NIXL agent: agent1 You can also run a complete Python example to test the installation: python3 examples/python/expanded two peers.py --mode=target --use cuda=true --ip=127.0.0.1 --port=4242 & sleep 5 python3 examples/python/expanded two peers.py --mode=initiator --use cuda=true --ip=127.0.0.1 --port=4242 For more Python examples, see examples/python/ https://github.com/ai-dynamo/nixl/blob/main/examples/python . - Use -Drust=true meson option to build rust bindings. - Use --buildtype=debug for a debug build default is release . - Or build manually: bash $ cargo build --release The bindings will be installed under nixl-sys in the configured installation prefix. Can be done using ninja, from project build directory: bash $ ninja install bash Rust bindings tests $ cargo test Use in your project by adding to Cargo.toml : dependencies nixl-sys = { path = "path/to/nixl/bindings/rust" } See contrib/README.md https://github.com/ai-dynamo/nixl/blob/main/contrib/README.md for more build options. To build the docker container, first clone the current repository. Also make sure you are able to pull docker images to your machine before attempting to build the container. Run the following from the root folder of the cloned NIXL repository: $ ./contrib/build-container.sh By default, the container is built with Ubuntu 24.04. To build a container for Ubuntu 22.04 use the --os option as follows: $ ./contrib/build-container.sh --os ubuntu22 To see all the options supported by the container use: $ ./contrib/build-container.sh -h The container has the NIXL python bindings preinstalled built from source against the container's own PyTorch . For a redistributable python wheel, use the wheel build script below or install the published nixl package. The contrib folder also includes a script to build the python wheel with the UCX dependencies. Note, that UCX and other NIXL dependencies are required to be installed. $ ./contrib/build-wheel.sh NIXL can use ETCD for metadata exchange between distributed nodes. This is especially useful in containerized or cloud-native environments. To use ETCD with NIXL, set the following environment variables: Set ETCD endpoints required - replace localhost with the hostname of the etcd server export NIXL ETCD ENDPOINTS="http://localhost:2379" Set ETCD namespace optional, defaults to /nixl/agents export NIXL ETCD NAMESPACE="/nixl/agents" NIXL includes an example demonstrating metadata exchange and data transfer using ETCD: Start an ETCD server if not already running For example: docker run -d -p 2379:2379 quay.io/coreos/etcd:v3.5.1 Set the ETCD env variables as above Run the example. The two agents in the example will exchange metadata through ETCD and perform data transfers ./<nixl build path /examples/nixl etcd example For more comprehensive testing, the nixlbench benchmarking tool supports ETCD for worker coordination: Build nixlbench see benchmark/nixlbench/README.md for details cd benchmark/nixlbench meson setup build && cd build && ninja Run benchmark with ETCD ./nixlbench --etcd-endpoints http://localhost:2379 --backend UCX --initiator seg type VRAM For contribution guidelines, see CONTRIBUTING.md https://github.com/ai-dynamo/nixl/blob/main/CONTRIBUTING.md CONTRIBUTING.md . This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use. NIXL Python wheels bundle NVIDIA modules libuct ib mlx5 ext.so , libuct ib mlx5 gda.so , libuct ib mlx5 gdp.so licensed under the NVIDIA Proprietary License https://github.com/ai-dynamo/nixl/blob/main/licenses/NVIDIA-proprietary-LICENSE.txt LicenseRef-NvidiaProprietary .