Run RF-DETR in NVIDIA DeepStream on Jetson

Roboflow published a guide showing how to run RF-DETR inside NVIDIA DeepStream on a Jetson Orin NX, using JetPack 6.2, DeepStream 7.1, and TensorRT 10.3. The post details exporting RF-DETR Nano to ONNX, building a TensorRT engine, and writing a custom parser to interpret the model's output tensors for object detection on RTSP streams.

This post shows how to run RF-DETR https://rfdetr.roboflow.com/latest/?ref=blog.roboflow.com inside NVIDIA DeepStream https://developer.nvidia.com/deepstream-sdk?ref=blog.roboflow.com on a Jetson Orin NX. We cover what DeepStream and RF-DETR are, how to build a TensorRT engine and the parser DeepStream needs to read the model's output, how to run it on RTSP stream, and how to give each class its own box color. We used NVIDIA's deepstream-import-vision-model https://github.com/NVIDIA/DeepStream/tree/main/skills/deepstream-import-vision-model?ref=blog.roboflow.com agent skill to do most of the import for us. Every number below was measured on JetPack 6.2 with DeepStream 7.1 and TensorRT 10.3. What Is DeepStream? DeepStream is NVIDIA's video analytics SDK, built on GStreamer https://gstreamer.freedesktop.org/?ref=blog.roboflow.com . It gives you hardware-accelerated pipeline parts decode, batching, TensorRT inference, on-screen display, encode that keep frames in GPU memory the whole way through. The ones you will touch in this post: nvstreammux : batches N decoded streams into one GPU buffer nvinfer : runs a TensorRT engine and attaches detection metadata to each frame nvdsosd : draws boxes and labels from that metadata Keep in mind the inference engine doesn't know or care what your model's output format is; it hands you raw tensors and leaves the interpretation to you. That interpretation lives in a small parser function you write, and it's where most of these projects succeed or fail. What Is RF-DETR? RF-DETR GitHub https://github.com/roboflow/rf-detr?ref=blog.roboflow.com by Roboflow https://roboflow.com/?ref=blog.roboflow.com is a DETR-family detector, released under Apache 2.0. It uses 300 learned queries that look over the image features, and each query predicts one box on its own. So there is no anchor grid and no NMS step afterward. It ships in a few sizes; here I'll use RF-DETR Nano, which takes a 384x384 RGB image and puts out two tensors: 300 boxes center x, center y, width, height, all scaled 0 to 1 and 300 x 91 class scores run through a sigmoid. Getting the Weights and ONNX The rfdetr package https://pypi.org/project/rfdetr/?ref=blog.roboflow.com downloads pretrained COCO weights the first time you use it, and exports a ready-to-deploy ONNX file for you: python pip install rfdetr from rfdetr import RFDETRNano model = RFDETRNano downloads pretrained weights model.export writes an ONNX file output dir=".", batch size=1, static batch, baked into the graph The export is already simplified and uses a fixed batch size by default, which matters on Jetson. Building the TensorRT engine is one command about two minutes on the Orin NX : /usr/src/tensorrt/bin/trtexec \ --onnx=rfdetr-nano.onnx \ --fp16 \ --saveEngine=rfdetr nano b1.engine Telling DeepStream What the Output Tensors Mean DeepStream doesn't know the output bytes are boxes, or how they are laid out, so you give it a parser function. DeepStream calls it with the raw output tensors for each frame, and it fills in detection structs class id, left, top, width, height, confidence . For RF-DETR, the post-processing steps are: run the class scores through a sigmoid, pick the best class for each query, and turn the center-form box into a corner-form box in pixels. Here is the core of it: // called by DeepStream with the raw output tensors of one frame extern "C" bool NvDsInferParseCustomRfdetr ... { // find the two output layers by name // "dets"/"labels" from the rfdetr export const float boxes = layer "dets" ; // 300, 4 cx, cy, w, h in 0,1 const float logits = layer "labels" ; // 300, 91 raw class logits for int q = 0; q < 300; q++ { // best class for this query skip index 0, the background slot int best = argmax logits + q 91 ; float score = sigmoid logits q 91 + best ; if score < threshold continue; NvDsInferObjectDetectionInfo obj = {}; // zero-init, always obj.classId = best; obj.detectionConfidence = score; // normalized center-form to pixel corner-form obj.width = boxes q 4 + 2 netW; obj.height = boxes q 4 + 3 netH; obj.left = boxes q 4 + 0 netW - obj.width / 2; obj.top = boxes q 4 + 1 netH - obj.height / 2; objectList.push back obj ; } return true; } The parser gets compiled into a small shared library, and the config file tells the system where it is, plus it holds the preprocessing settings. But if you mess any of it up, you won't get any notification that something's wrong - you'll either see no detections or boxes in totally wrong spots. property onnx-file=rfdetr-nano.onnx model-engine-file=rfdetr nano b1.engine labelfile-path=labels.txt 91 COCO class names, one per line RF-DETR expects plain 0..1 input: rescale only, no ImageNet normalization net-scale-factor=0.00392156862745098 = 1/255 model-color-format=0 RGB maintain-aspect-ratio=0 model squash-resizes to 384x384, no letterbox cluster-mode=4 DETR family: no NMS network-mode=2 FP16 infer-dims=3;384;384 num-detected-classes=91 custom-lib-path=libnvdsinfer rfdetr parser.so parse-bbox-func-name=NvDsInferParseCustomRfdetr class-attrs-all pre-cluster-threshold=0.4 confidence cutoff Check the model's preprocessing settings instead of assuming ImageNet normalization. This export expects plain 0 to 1 input. Running on a Live Camera Once the engine, parser, and config are in place, running a live camera takes a single deepstream-app config file: application enable-perf-measurement=1 prints per-stream FPS every few seconds source0 enable=1 type=4 4 = RTSP source uri=rtsp://user:pass@CAMERA IP:554/stream1 latency=200 jitter buffer, ms streammux batch-size=1 match the engine's batch size width=1280 frames are scaled to this before inference height=720 live-source=1 primary-gie enable=1 config-file=config infer rfdetr.txt the nvinfer config above sink0 enable=1 type=1 1 = fakesink headless ; use type=4 to re-stream sync=0 deepstream-app -c ds app rtsp.txt A single 720p stream runs in real time with plenty of headroom on the Orin NX. If you want to record or re-stream the annotated output, the Orin NX has a hardware H.264 encoder NVENC , so encoding stays on the GPU nvv4l2h264enc instead of loading the CPU. Per-Class Colors Need Code, Not Config There is no config key for coloring boxes by class. The color lives in per-object metadata, and you change it in a probe a callback on the pipeline right before the on-screen display draws the box. That means writing a small app of your own instead of using the stock binary. The Python bindings https://github.com/NVIDIA-AI-IOT/deepstream python apps?ref=blog.roboflow.com install as a wheel, with no root needed, and the whole thing is about 20 lines: python import colorsys, pyds from gi.repository import Gst 91 visually distinct hues via golden-ratio spacing PALETTE = colorsys.hsv to rgb i 0.618 % 1.0, 0.85, 1.0 for i in range 91 def osd probe pad, info, : batch = pyds.gst buffer get nvds batch meta hash info.get buffer l frame = batch.frame meta list while l frame: frame = pyds.NvDsFrameMeta.cast l frame.data l obj = frame.obj meta list while l obj: obj = pyds.NvDsObjectMeta.cast l obj.data r, g, b = PALETTE obj.class id % 91 obj.rect params.border width = 3 obj.rect params.border color.set r, g, b, 1.0 box color obj.text params.set bg clr = 1 obj.text params.text bg clr.set r, g, b, 0.55 label background l obj = l obj.next l frame = l frame.next return Gst.PadProbeReturn.OK attach to the on-screen-display element's input osd.get static pad "sink" .add probe Gst.PadProbeType.BUFFER, osd probe, 0 Attach that to the same pipeline, built in Python, and every class gets its own steady color, with the label background matched to the box. Four streams batched through one RF-DETR Nano engine on a Jetson Orin NX, tiled 2x2 with the colored-box probe above, hardware-encoded and streamed off the device. Every detection here is the same class car , so they share one color; the probe assigns a distinct hue per class id when the scene has more than one. Measured Throughput Numbers from this setup: RF-DETR Nano, FP16, 384x384, on an Orin NX 16GB in MAXN mode. The TensorRT rows are trtexec on the engine alone; the DeepStream row is the full four-stream pipeline decode, inference, on-screen display, and H.264 encode , with frames counted on the inference output. | Metric | Orin NX 16GB Nano, FP16, 384 | |---|---| | TensorRT, batch 1 | 132 img/s 7.6 ms latency | | TensorRT, batch 4 | 160 img/s 25 ms/batch | | DeepStream, 4x 720p RTSP at 30 fps | 30 fps/stream sustained 120 aggregate | | GPU utilization, 4-stream pipeline | ~97% | | Peak RAM, 4-stream pipeline | 3.6 GB | Batching from 1 to 4 lifts inference throughput about 20% 132 to 160 img/s , less than the gains you see with CNN detectors, because a detection transformer already keeps the GPU busy at batch 1. And the four live streams hold their full 30 fps with the GPU pinned near 97%: pure inference tops out around 160 img/s, and the rest of the budget goes to decoding four streams, tiling, drawing boxes, and H.264 encoding, all sharing the same GPU. So there's basically no room left to add a fifth 30 fps stream with this model on this device. Conclusion We've done the full path from pretrained weights to live camera inference on a Jetson: export ONNX with the rfdetr package, build an FP16 TRT engine, add the parser that tells DeepStream what the tensors mean, point deepstream-app at your camera, and add a Python probe when you want per-class colors. The DeepStream import skill https://github.com/NVIDIA/DeepStream/tree/main/skills/deepstream-import-vision-model?ref=blog.roboflow.com does most of this for you. RF-DETR is on GitHub https://github.com/roboflow/rf-detr?ref=blog.roboflow.com under Apache 2.0. If you want to train one on your own data, the Custom Model Training guide https://docs.roboflow.com/guides/model-training?ref=blog.roboflow.com covers that. Cite this Post Use the following entry to cite this post in your research: Erik Kokalj /author/erik/ . Jul 22, 2026 . Run RF-DETR in NVIDIA DeepStream on Jetson. Roboflow Blog: https://blog.roboflow.com/rf-detr-nvidia-deepstream/