{"slug": "run-native-dla-inference-in-python-on-nvidia-jetson-agx-orin-using-cudla", "title": "Run Native DLA Inference in Python on NVIDIA Jetson AGX Orin Using cuDLA", "summary": "NVIDIA released cuda.bindings.cudla in cuda-bindings 13.3.0, a Cython-generated Python layer over the cuDLA C++ API that lets developers run Deep Learning Accelerator inference on Jetson AGX Orin without writing C++. The module exposes thirteen functions, nine data types and eight enums for hybrid-mode workflows in which CUDA streams coordinate DLA execution with GPU work, and NVIDIA said the identical API also applies to its RTX Spark Windows AI developer laptops. On Jetson Orin, JetPack 7.2 supplies the underlying libcudla.so runtime, so no separate cuDLA SDK is required, and the bindings need Python 3.10 or later.", "body_md": "AI applications at the edge often need to run multiple neural networks within limited compute and power budgets. NVIDIA [Deep Learning Accelerator (DLA)](https://developer.nvidia.com/deep-learning-accelerator) platforms provide dedicated hardware for deep learning inference. On [NVIDIA Jetson AGX Orin](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/), DLA executes supported neural network operations, allowing applications to offload inference work from the GPU. This can improve power efficiency and leave GPU capacity available for other application tasks.\n\nNVIDIA cuDLA is the runtime API for managing execution on this accelerator. Applications use it to create DLA devices, load compiled networks, register memory buffers, and submit inference tasks. The cuda.bindings.cudla module exposes these APIs directly to Python, so developers can integrate DLA execution into their applications without writing C++ code to manage the runtime.\n\nThis post walks through the inference lifecycle on Jetson AGX Orin using an image classification example. It briefly covers installation and compiling a model into a DLA loadable, then follows execution from creating a device and loading the network to managing buffers, synchronizing execution, and reading the output. The example also shows how GPU and DLA stages work together in a single Python application.\n\n## **How cuDLA fits within a Python application**\n\nDLA is a fixed-function accelerator. This means that it efficiently performs fixed operations, such as convolutions and pooling, that are common in modern neural network architectures. It does not have as many supported layers as the GPU, but it still supports a wide variety of layers used in many popular neural network architectures.. To learn more about when and why to use DLA, see [Getting Started with the Deep Learning Accelerator on NVIDIA Jetson Orin](https://developer.nvidia.com/blog/getting-started-with-the-deep-learning-accelerator-on-nvidia-jetson-orin/).\n\nIntroduced in cuda-bindings 13.3.0 as part of the [CUDA Python](https://github.com/NVIDIA/cuda-python) project, cuda.bindings.cudla is a thin, Cython-generated Python layer over the cuDLA C++ API for hybrid-mode workflows. It exposes thirteen functions, nine data types, eight enums for hybrid-mode workflows, where CUDA streams coordinate DLA execution with GPU work. On Jetson Orin, [NVIDIA JetPack](https://developer.nvidia.com/embedded/jetpack) supplies the underlying libcudla.so runtime. \n\nThe Python bindings expose that runtime directly to your application, while inference executes on DLA hardware. This complements the existing C++ workflow by allowing runtime management and inference orchestration to remain in Python. Figure 1 shows where cuda.bindings.cudla fits in the stack.\n\nWhile this post focuses on Jetson, [NVIDIA RTX Spark](https://www.nvidia.com/en-us/products/rtx-spark/) brings DLA inference to mainstream Windows AI developer laptops for the first time, extending cuDLA well beyond the embedded and robotics space. The cuda.bindings.cudla API is identical on both platforms, so everything in this post related to cuDLA APIs applies directly to RTX Spark as well.\n\n## **Prerequisites and installation**\n\nIf you are on an NVIDIA Jetson Orin running [JetPack](https://developer.nvidia.com/embedded/jetpack/downloads/archive-7.2), libcudla.so ships with the JetPack installation and is automatically discoverable by the bindings. There is nothing additional to install for the DLA runtime itself—no separate cuDLA SDK and no extra runtime libraries.\n\nInstall the cuDLA Python bindings:\n\n```\npip install \"cuda-bindings>=13.3.0\"\n```\n\nThe JetPack default Python may be older than 3.10, which is the minimum required version. Install Python 3.12 from the deadsnakes PPA if needed:\n\n```\nsudo apt update && sudo apt install -y software-properties-common\nsudo add-apt-repository -y ppa:deadsnakes/ppa\nsudo apt update && sudo apt install -y python3.12 python3.12-venv python3.12-dev\npython3.12 -m venv ~/dla-env\n~/dla-env/bin/pip install \"cuda-bindings>=13.3.0\"\n```\n\nYou only need TensorRT to *compile* a trained model into a DLA loadable. To *run* an existing loadable, you only need `libcudla.so`, which ships with JetPack.\n\n## **Compiling a DLA loadable**\n\nA DLA loadable is the binary artifact produced by the TensorRT trtexec tool. It encodes a trained network in a format DLA hardware can execute directly. This compilation example targets TensorRT 11.4.1 with CUDA 13.4 Update 1 on Jetson AGX Orin running JetPack 7.2.2 or later. For an FP16 CHW16 loadable, using an FP16-typed ONNX model with fixed input dimensions, use the following:\n\n```\ntrtexec \\\n    --onnx=your_model_fp16.onnx \\\n    --useDLACore=0 \\\n    --inputIOFormats=chw16 \\\n    --outputIOFormats=chw16 \\\n    --memPoolSize=dlaSRAM:1 \\\n    --saveEngine=model.bin \\\n    --buildDLAStandalone\n```\n\nThe key flags are:\n\n- `--useDLACore=0` targets the first DLA engine on Orin\n- `--buildDLAStandalone` produces a raw DLA loadable that does not require TensorRT at inference time\n- I/O format options select CHW16 layout\n\nTensorRT 11 derives precision from the model’s tensor types; `--fp16` and `--int8` are no longer supported. This example assumes fixed input dimensions, so `--shapes` is omitted.\n\nThis reflects the sample’s current FP16 path and [TensorRT 11 flag changes](https://docs.nvidia.com/deeplearning/tensorrt/latest/api/migration/tensorrt-10x-to-11x-trtexec.html).\n\nNot every network layer is DLA-compatible, and preprocessing the ONNX graph to trim DLA-incompatible head and tail layers is often necessary. The Python workflow in the next section works with any .bin or .nvdla file you produce. Figure 2 shows the split between the one-time build step and the on-device runtime.\n\n## **The Python workflow**\n\nFor the CUDA and cuDLA calls, three imports cover everything:\n\n``` python\nimport numpy as np\nfrom cuda.bindings import cudla\nfrom cuda.bindings import driver, runtime\n```\n\n`cudla` is the new module. `driver` and `runtime` are the existing cuda-python bindings for the CUDA driver and runtime APIs—used here for stream management, memory allocation, and data transfers. Everything runs through a single CUDA stream, the coordination point for DLA and GPU work in hybrid mode.\n\n### **Step 1: Initialize CUDA and create a DLA device**\n\n`cudla.Mode.CUDA_DLA` selects hybrid mode, where you submit DLA tasks through a CUDA stream. This enables GPU-DLA pipelines: preprocessing kernels, DLA inference, and postprocessing kernels can all be enqueued on the same stream and executed in order.\n\n```\ndriver.cuInit(0)\nruntime.cudaSetDevice(0)\n\ncudla_device = cudla.create_device(0, int(cudla.Mode.CUDA_DLA))\n```\n\nThis release does not support standalone mode (which requires the NvSci buffer and sync primitives). `create_device` raises `CudlaError(ErrorUnsupportedOperation)` if you request it.\n\n### **Step 2: Load the loadable**\n\nPass the loadable as a Python `bytes` object directly to `module_load_from_memory`. No file path or filesystem handle—just raw bytes, equally easy to load from memory-mapped files, embedded resources, or network streams.\n\n```\nwith open(\"model.bin\", \"rb\") as f:\n    loadable_data = f.read()\n\nmodule_handle = cudla.module_load_from_memory(\n    cudla_device, loadable_data, len(loadable_data), 0\n)\n```\n\n### **Step 3: Introspect the loaded module**\n\nBefore allocating any memory, query the module to find out exactly what tensors it expects:\n\n```\nnum_inputs = cudla.module_get_attributes(\n    module_handle, int(cudla.ModuleAttributeType.NUM_INPUT_TENSORS)\n)\nnum_outputs = cudla.module_get_attributes(\n    module_handle, int(cudla.ModuleAttributeType.NUM_OUTPUT_TENSORS)\n)\ninput_descs = cudla.module_get_attributes(\n    module_handle, int(cudla.ModuleAttributeType.INPUT_TENSOR_DESCRIPTORS)\n)\noutput_descs = cudla.module_get_attributes(\n    module_handle, int(cudla.ModuleAttributeType.OUTPUT_TENSOR_DESCRIPTORS)\n)\n```\n\nEach descriptor is a `cudla.ModuleTensorDescriptor` with properties for buffer size (`size_`), tensor shape (` n, c, h, w`), name, data format, and stride. All subsequent allocation calls are driven by `desc.size_` with no hardcoded sizes anywhere in the application. This introspection step is what makes the workflow model-agnostic: the same code paths handle any loadable regardless of network architecture.\n\n### **Step 4: Allocate GPU memory**\n\n```\ninput_gpu  = runtime.cudaMalloc(input_descs[0].size_)[1]\noutput_gpu = runtime.cudaMalloc(output_descs[0].size_)[1]\n```\n\nCUDA driver and runtime calls return a status code as the first element of a tuple. The full sample wraps every such call in a helper that checks the returned status and raises on failure—the snippets here elide that for readability, but error checking is essential in production code. For models with multiple inputs or outputs, iterate over `input_descs` and `output_descs` and allocate one buffer per tensor. The `size_` field on each descriptor gives the exact byte count.\n\n### **Step 5: Register GPU buffers with cuDLA**\n\nThis is the step unique to cuDLA. GPU buffers must be registered with the DLA device before they can appear in a task. Registration returns a DLA-side handle that the hardware uses to access the buffer:\n\n```\ninput_registered  = cudla.mem_register(\n    cudla_device, int(input_gpu), input_descs[0].size_, 0\n)\noutput_registered = cudla.mem_register(\n    cudla_device, int(output_gpu), output_descs[0].size_, 0\n)\n```\n\nThe `int()` cast extracts the raw pointer from the CUDA memory object. `cuda.bindings` returns opaque handle objects, and `int()` is how you pass them to C-level APIs that expect a pointer.\n\n### **Step 6: Prepare input data and copy to GPU**\n\nPreprocessing—normalizing values, matching the expected tensor stride and data format—is the only model-specific code in the script. The tensor descriptor `stride`, `data_format`, and `data_type` fields tell you exactly how to lay out the data. For a multi-input model, repeat the copy for each input buffer. Everything else before and after this block is identical regardless of what network is inside the loadable.\n\n```\nstream = driver.cuStreamCreate(driver.CUstream_flags.CU_STREAM_NON_BLOCKING)[1]\n\n# Preprocess your input into a numpy array `input_cpu`\n# matching the tensor layout described by input_descs[0] ...\n\nruntime.cudaMemcpyAsync(\n    input_gpu, input_cpu, input_descs[0].size_,\n    runtime.cudaMemcpyKind.cudaMemcpyHostToDevice, stream\n)\nruntime.cudaMemsetAsync(output_gpu, 0, output_descs[0].size_, stream)\n```\n\n### **Step 7: Build the task and submit**\n\n`cudla.Task` wraps the `cudlaTask` C struct. Assigning a Python list to `task.input_tensor` or `task.output_tensor` automatically updates the corresponding count field. `submit_task` enqueues the inference work onto the CUDA stream, sequencing it with any GPU kernels before or after it on the same stream.\n\n```\ntask = cudla.Task()\ntask.module_handle = module_handle\ntask.input_tensor  = [int(input_registered)]\ntask.output_tensor = [int(output_registered)]\ntask.wait_events   = 0\ntask.signal_events = 0\n\ncudla.submit_task(cudla_device, int(task), 1, int(stream), 0)\n```\n\n### **Step 8: Synchronize and read back**\n\nThe first sync waits for the DLA task to complete. Allocating `output_cpu` as a numpy array of the descriptor’s byte size gives the D2H copy a destination buffer; the second sync ensures the transfer is done before you read from it.\n\n```\nruntime.cudaStreamSynchronize(stream)\n\noutput_cpu = np.empty(output_descs[0].size_, dtype=np.uint8)\nruntime.cudaMemcpyAsync(\n    output_cpu, output_gpu, output_descs[0].size_,\n    runtime.cudaMemcpyKind.cudaMemcpyDeviceToHost, stream\n)\nruntime.cudaStreamSynchronize(stream)\n```\n\n### **Step 9: Cleanup**\n\nEvery handle created has a matching destroy call—DLA device, module, registered memory handles, GPU allocations, and stream.\n\n```\ncudla.mem_unregister(cudla_device, input_registered)\ncudla.mem_unregister(cudla_device, output_registered)\ncudla.module_unload(module_handle, 0)\ncudla.destroy_device(cudla_device)\nruntime.cudaFree(input_gpu)\nruntime.cudaFree(output_gpu)\ndriver.cuStreamDestroy(stream)\n```\n\n## **Error handling is native to Python**\n\nAll cuDLA API calls raise `cudla.CudlaError` on failure rather than returning error codes. The exception carries both the numeric status and its human-readable name:\n\n```\ntry:\n    device = cudla.create_device(0, int(cudla.Mode.CUDA_DLA))\nexcept cudla.CudlaError as e:\n    print(e)  # e.g. \"ErrorInvalidDevice (8)\"\n```\n\nDLA workflows fit naturally into standard Python exception handling. I recommend `try/finally` for cleanup. DLA device handles are opaque C resources and will not release themselves on garbage collection. Branch on `e.status` against `cudla.Status` enum values to handle specific failure modes, from parameter errors (`ErrorInvalidParam`) up to hardware-level faults (` ErrorDlaErrEngineTimeout`).\n\n## **What is the resulting output?** \n\nAfter `cudaStreamSynchronize` returns, `output_cpu` holds the raw DLA output for your network as a numpy array, ready for any standard Python postprocessing. What you do with it depends entirely on the model, including class scores, feature maps, bounding box coordinates. The cuDLA Python layer hands you the bytes. Interpreting them is the same postprocessing you would write in any Python inference pipeline, and you can chain it directly with NumPy, PyTorch, or OpenCV without leaving the Python process.\n\nTo make this concrete: for a ResNet-50 backbone compiled as a DLA loadable, the sample output would look like this:\n\n```\ncuDLA device created: 0x55a1b2c30000\nModule loaded: 0x55a5b6c70000\nnumInputTensors = 1\nnumOutputTensors = 1\nInput 0:  size=..., shape=(1,64,56,56), name=resnetv17_pool0_fwd\nOutput 0: size=..., shape=(1,2048,7,7), name=resnetv17_stage4_activation2\nSUBMIT IS DONE (task queued; completion pending)\nOutput tensor written to host\n```\n\nThe sample output tells you the exact buffer sizes and tensor names the loadable expects, which your preprocessing and postprocessing code can read directly rather than hardcoding them. After synchronization, `output_cpu` is a flat numpy array; interpret or reshape it using the `data_type`, `data_format`, `stride`, and shape fields from `output_descs[0]` and pass it to whatever downstream stage your pipeline requires.\n\nThe workflow from `create_device` to `cudaStreamSynchronize` is the same regardless of network architecture. Only the tensor names, sizes, and postprocessing change.\n\n## Final steps\n\nThe `cuda.bindings.cudla` module ships in `cuda-bindings` 13.3.0. Install it, compile a loadable with `trtexec`, and the nine steps above take you from a `.bin` file to DLA inference results directly in Python on the same board.\n\n```\npip install \"cuda-bindings>=13.3.0\"\n```\n\nThe unit tests for enums, POD types, and API surface ship in the [NVIDIA/cuda-python](https://github.com/NVIDIA/cuda-python) GitHub repo under `cuda_bindings/tests/cudla/` and run entirely without DLA hardware—no `libcudla.so` required:\n\n```\npytest cuda_bindings/tests/cudla/ -v\n```\n\n## Get started with cuDLA in Python\n\nFor a complete end-to-end inference example, sample inputs, and scripts to prepare the model and compile a DLA loadable locally, refer to the [Deep Learning Accelerator samples](https://github.com/NVIDIA/Deep-Learning-Accelerator-SW/tree/main/samples/cuDLA/cuDLAPythonE2E).\n\nTo learn more, visit the [NVIDIA/cuda-python](https://github.com/NVIDIA/cuda-python) GitHub repo.", "url": "https://wpnews.pro/news/run-native-dla-inference-in-python-on-nvidia-jetson-agx-orin-using-cudla", "canonical_source": "https://developer.nvidia.com/blog/run-native-dla-inference-in-python-on-nvidia-jetson-agx-orin-using-cudla/", "published_at": "2026-11-13 18:57:10+00:00", "updated_at": "2026-11-13 19:19:20.076783+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-chips", "developer-tools", "ai-tools"], "entities": ["NVIDIA", "Jetson AGX Orin", "cuDLA", "cuda.bindings.cudla", "CUDA Python", "JetPack", "RTX Spark", "Deep Learning Accelerator"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/run-native-dla-inference-in-python-on-nvidia-jetson-agx-orin-using-cudla", "markdown": "https://wpnews.pro/news/run-native-dla-inference-in-python-on-nvidia-jetson-agx-orin-using-cudla.md", "text": "https://wpnews.pro/news/run-native-dla-inference-in-python-on-nvidia-jetson-agx-orin-using-cudla.txt", "jsonld": "https://wpnews.pro/news/run-native-dla-inference-in-python-on-nvidia-jetson-agx-orin-using-cudla.jsonld"}}