Run Native DLA Inference in Python on NVIDIA Jetson AGX Orin Using cuDLA

NVIDIA released cuda.bindings.cudla in cuda-bindings 13.3.0, a Cython-generated Python layer over the cuDLA C++ API that lets developers run Deep Learning Accelerator inference on Jetson AGX Orin without writing C++. The module exposes thirteen functions, nine data types and eight enums for hybrid-mode workflows in which CUDA streams coordinate DLA execution with GPU work, and NVIDIA said the identical API also applies to its RTX Spark Windows AI developer laptops. On Jetson Orin, JetPack 7.2 supplies the underlying libcudla.so runtime, so no separate cuDLA SDK is required, and the bindings need Python 3.10 or later.

AI applications at the edge often need to run multiple neural networks within limited compute and power budgets. NVIDIA Deep Learning Accelerator DLA https://developer.nvidia.com/deep-learning-accelerator platforms provide dedicated hardware for deep learning inference. On NVIDIA Jetson AGX Orin https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/ , DLA executes supported neural network operations, allowing applications to offload inference work from the GPU. This can improve power efficiency and leave GPU capacity available for other application tasks. NVIDIA cuDLA is the runtime API for managing execution on this accelerator. Applications use it to create DLA devices, load compiled networks, register memory buffers, and submit inference tasks. The cuda.bindings.cudla module exposes these APIs directly to Python, so developers can integrate DLA execution into their applications without writing C++ code to manage the runtime. This post walks through the inference lifecycle on Jetson AGX Orin using an image classification example. It briefly covers installation and compiling a model into a DLA loadable, then follows execution from creating a device and loading the network to managing buffers, synchronizing execution, and reading the output. The example also shows how GPU and DLA stages work together in a single Python application. How cuDLA fits within a Python application DLA is a fixed-function accelerator. This means that it efficiently performs fixed operations, such as convolutions and pooling, that are common in modern neural network architectures. It does not have as many supported layers as the GPU, but it still supports a wide variety of layers used in many popular neural network architectures.. To learn more about when and why to use DLA, see Getting Started with the Deep Learning Accelerator on NVIDIA Jetson Orin https://developer.nvidia.com/blog/getting-started-with-the-deep-learning-accelerator-on-nvidia-jetson-orin/ . Introduced in cuda-bindings 13.3.0 as part of the CUDA Python https://github.com/NVIDIA/cuda-python project, cuda.bindings.cudla is a thin, Cython-generated Python layer over the cuDLA C++ API for hybrid-mode workflows. It exposes thirteen functions, nine data types, eight enums for hybrid-mode workflows, where CUDA streams coordinate DLA execution with GPU work. On Jetson Orin, NVIDIA JetPack https://developer.nvidia.com/embedded/jetpack supplies the underlying libcudla.so runtime. The Python bindings expose that runtime directly to your application, while inference executes on DLA hardware. This complements the existing C++ workflow by allowing runtime management and inference orchestration to remain in Python. Figure 1 shows where cuda.bindings.cudla fits in the stack. While this post focuses on Jetson, NVIDIA RTX Spark https://www.nvidia.com/en-us/products/rtx-spark/ brings DLA inference to mainstream Windows AI developer laptops for the first time, extending cuDLA well beyond the embedded and robotics space. The cuda.bindings.cudla API is identical on both platforms, so everything in this post related to cuDLA APIs applies directly to RTX Spark as well. Prerequisites and installation If you are on an NVIDIA Jetson Orin running JetPack https://developer.nvidia.com/embedded/jetpack/downloads/archive-7.2 , libcudla.so ships with the JetPack installation and is automatically discoverable by the bindings. There is nothing additional to install for the DLA runtime itself—no separate cuDLA SDK and no extra runtime libraries. Install the cuDLA Python bindings: pip install "cuda-bindings =13.3.0" The JetPack default Python may be older than 3.10, which is the minimum required version. Install Python 3.12 from the deadsnakes PPA if needed: sudo apt update && sudo apt install -y software-properties-common sudo add-apt-repository -y ppa:deadsnakes/ppa sudo apt update && sudo apt install -y python3.12 python3.12-venv python3.12-dev python3.12 -m venv ~/dla-env ~/dla-env/bin/pip install "cuda-bindings =13.3.0" You only need TensorRT to compile a trained model into a DLA loadable. To run an existing loadable, you only need libcudla.so , which ships with JetPack. Compiling a DLA loadable A DLA loadable is the binary artifact produced by the TensorRT trtexec tool. It encodes a trained network in a format DLA hardware can execute directly. This compilation example targets TensorRT 11.4.1 with CUDA 13.4 Update 1 on Jetson AGX Orin running JetPack 7.2.2 or later. For an FP16 CHW16 loadable, using an FP16-typed ONNX model with fixed input dimensions, use the following: trtexec \ --onnx=your model fp16.onnx \ --useDLACore=0 \ --inputIOFormats=chw16 \ --outputIOFormats=chw16 \ --memPoolSize=dlaSRAM:1 \ --saveEngine=model.bin \ --buildDLAStandalone The key flags are: - --useDLACore=0 targets the first DLA engine on Orin - --buildDLAStandalone produces a raw DLA loadable that does not require TensorRT at inference time - I/O format options select CHW16 layout TensorRT 11 derives precision from the model’s tensor types; --fp16 and --int8 are no longer supported. This example assumes fixed input dimensions, so --shapes is omitted. This reflects the sample’s current FP16 path and TensorRT 11 flag changes https://docs.nvidia.com/deeplearning/tensorrt/latest/api/migration/tensorrt-10x-to-11x-trtexec.html . Not every network layer is DLA-compatible, and preprocessing the ONNX graph to trim DLA-incompatible head and tail layers is often necessary. The Python workflow in the next section works with any .bin or .nvdla file you produce. Figure 2 shows the split between the one-time build step and the on-device runtime. The Python workflow For the CUDA and cuDLA calls, three imports cover everything: python import numpy as np from cuda.bindings import cudla from cuda.bindings import driver, runtime cudla is the new module. driver and runtime are the existing cuda-python bindings for the CUDA driver and runtime APIs—used here for stream management, memory allocation, and data transfers. Everything runs through a single CUDA stream, the coordination point for DLA and GPU work in hybrid mode. Step 1: Initialize CUDA and create a DLA device cudla.Mode.CUDA DLA selects hybrid mode, where you submit DLA tasks through a CUDA stream. This enables GPU-DLA pipelines: preprocessing kernels, DLA inference, and postprocessing kernels can all be enqueued on the same stream and executed in order. driver.cuInit 0 runtime.cudaSetDevice 0 cudla device = cudla.create device 0, int cudla.Mode.CUDA DLA This release does not support standalone mode which requires the NvSci buffer and sync primitives . create device raises CudlaError ErrorUnsupportedOperation if you request it. Step 2: Load the loadable Pass the loadable as a Python bytes object directly to module load from memory . No file path or filesystem handle—just raw bytes, equally easy to load from memory-mapped files, embedded resources, or network streams. with open "model.bin", "rb" as f: loadable data = f.read module handle = cudla.module load from memory cudla device, loadable data, len loadable data , 0 Step 3: Introspect the loaded module Before allocating any memory, query the module to find out exactly what tensors it expects: num inputs = cudla.module get attributes module handle, int cudla.ModuleAttributeType.NUM INPUT TENSORS num outputs = cudla.module get attributes module handle, int cudla.ModuleAttributeType.NUM OUTPUT TENSORS input descs = cudla.module get attributes module handle, int cudla.ModuleAttributeType.INPUT TENSOR DESCRIPTORS output descs = cudla.module get attributes module handle, int cudla.ModuleAttributeType.OUTPUT TENSOR DESCRIPTORS Each descriptor is a cudla.ModuleTensorDescriptor with properties for buffer size size , tensor shape n, c, h, w , name, data format, and stride. All subsequent allocation calls are driven by desc.size with no hardcoded sizes anywhere in the application. This introspection step is what makes the workflow model-agnostic: the same code paths handle any loadable regardless of network architecture. Step 4: Allocate GPU memory input gpu = runtime.cudaMalloc input descs 0 .size 1 output gpu = runtime.cudaMalloc output descs 0 .size 1 CUDA driver and runtime calls return a status code as the first element of a tuple. The full sample wraps every such call in a helper that checks the returned status and raises on failure—the snippets here elide that for readability, but error checking is essential in production code. For models with multiple inputs or outputs, iterate over input descs and output descs and allocate one buffer per tensor. The size field on each descriptor gives the exact byte count. Step 5: Register GPU buffers with cuDLA This is the step unique to cuDLA. GPU buffers must be registered with the DLA device before they can appear in a task. Registration returns a DLA-side handle that the hardware uses to access the buffer: input registered = cudla.mem register cudla device, int input gpu , input descs 0 .size , 0 output registered = cudla.mem register cudla device, int output gpu , output descs 0 .size , 0 The int cast extracts the raw pointer from the CUDA memory object. cuda.bindings returns opaque handle objects, and int is how you pass them to C-level APIs that expect a pointer. Step 6: Prepare input data and copy to GPU Preprocessing—normalizing values, matching the expected tensor stride and data format—is the only model-specific code in the script. The tensor descriptor stride , data format , and data type fields tell you exactly how to lay out the data. For a multi-input model, repeat the copy for each input buffer. Everything else before and after this block is identical regardless of what network is inside the loadable. stream = driver.cuStreamCreate driver.CUstream flags.CU STREAM NON BLOCKING 1 Preprocess your input into a numpy array input cpu matching the tensor layout described by input descs 0 ... runtime.cudaMemcpyAsync input gpu, input cpu, input descs 0 .size , runtime.cudaMemcpyKind.cudaMemcpyHostToDevice, stream runtime.cudaMemsetAsync output gpu, 0, output descs 0 .size , stream Step 7: Build the task and submit cudla.Task wraps the cudlaTask C struct. Assigning a Python list to task.input tensor or task.output tensor automatically updates the corresponding count field. submit task enqueues the inference work onto the CUDA stream, sequencing it with any GPU kernels before or after it on the same stream. task = cudla.Task task.module handle = module handle task.input tensor = int input registered task.output tensor = int output registered task.wait events = 0 task.signal events = 0 cudla.submit task cudla device, int task , 1, int stream , 0 Step 8: Synchronize and read back The first sync waits for the DLA task to complete. Allocating output cpu as a numpy array of the descriptor’s byte size gives the D2H copy a destination buffer; the second sync ensures the transfer is done before you read from it. runtime.cudaStreamSynchronize stream output cpu = np.empty output descs 0 .size , dtype=np.uint8 runtime.cudaMemcpyAsync output cpu, output gpu, output descs 0 .size , runtime.cudaMemcpyKind.cudaMemcpyDeviceToHost, stream runtime.cudaStreamSynchronize stream Step 9: Cleanup Every handle created has a matching destroy call—DLA device, module, registered memory handles, GPU allocations, and stream. cudla.mem unregister cudla device, input registered cudla.mem unregister cudla device, output registered cudla.module unload module handle, 0 cudla.destroy device cudla device runtime.cudaFree input gpu runtime.cudaFree output gpu driver.cuStreamDestroy stream Error handling is native to Python All cuDLA API calls raise cudla.CudlaError on failure rather than returning error codes. The exception carries both the numeric status and its human-readable name: try: device = cudla.create device 0, int cudla.Mode.CUDA DLA except cudla.CudlaError as e: print e e.g. "ErrorInvalidDevice 8 " DLA workflows fit naturally into standard Python exception handling. I recommend try/finally for cleanup. DLA device handles are opaque C resources and will not release themselves on garbage collection. Branch on e.status against cudla.Status enum values to handle specific failure modes, from parameter errors ErrorInvalidParam up to hardware-level faults ErrorDlaErrEngineTimeout . What is the resulting output? After cudaStreamSynchronize returns, output cpu holds the raw DLA output for your network as a numpy array, ready for any standard Python postprocessing. What you do with it depends entirely on the model, including class scores, feature maps, bounding box coordinates. The cuDLA Python layer hands you the bytes. Interpreting them is the same postprocessing you would write in any Python inference pipeline, and you can chain it directly with NumPy, PyTorch, or OpenCV without leaving the Python process. To make this concrete: for a ResNet-50 backbone compiled as a DLA loadable, the sample output would look like this: cuDLA device created: 0x55a1b2c30000 Module loaded: 0x55a5b6c70000 numInputTensors = 1 numOutputTensors = 1 Input 0: size=..., shape= 1,64,56,56 , name=resnetv17 pool0 fwd Output 0: size=..., shape= 1,2048,7,7 , name=resnetv17 stage4 activation2 SUBMIT IS DONE task queued; completion pending Output tensor written to host The sample output tells you the exact buffer sizes and tensor names the loadable expects, which your preprocessing and postprocessing code can read directly rather than hardcoding them. After synchronization, output cpu is a flat numpy array; interpret or reshape it using the data type , data format , stride , and shape fields from output descs 0 and pass it to whatever downstream stage your pipeline requires. The workflow from create device to cudaStreamSynchronize is the same regardless of network architecture. Only the tensor names, sizes, and postprocessing change. Final steps The cuda.bindings.cudla module ships in cuda-bindings 13.3.0. Install it, compile a loadable with trtexec , and the nine steps above take you from a .bin file to DLA inference results directly in Python on the same board. pip install "cuda-bindings =13.3.0" The unit tests for enums, POD types, and API surface ship in the NVIDIA/cuda-python https://github.com/NVIDIA/cuda-python GitHub repo under cuda bindings/tests/cudla/ and run entirely without DLA hardware—no libcudla.so required: pytest cuda bindings/tests/cudla/ -v Get started with cuDLA in Python For a complete end-to-end inference example, sample inputs, and scripts to prepare the model and compile a DLA loadable locally, refer to the Deep Learning Accelerator samples https://github.com/NVIDIA/Deep-Learning-Accelerator-SW/tree/main/samples/cuDLA/cuDLAPythonE2E . To learn more, visit the NVIDIA/cuda-python https://github.com/NVIDIA/cuda-python GitHub repo.