TensorRT Was Fast. My 1080p RTSP Pipeline Wasn't.

I was troubleshooting low FPS in a TensorRT-based RTSP video analytics pipeline on a customer's server. The input was a 1920×1080 stream at 25 FPS, running on an NVIDIA GeForce RTX 5070 Ti. The pose model already used TensorRT, but the application was processing only 8–10 FPS.

Inference was accelerated. The video pipeline wasn't.

After changing how frames moved through the application, I measured 24.64 processed FPS against the same 25 FPS source. That measurement was taken before the OCR queue; it was not a benchmark of end-to-end OCR or video delivery latency.

I didn't replace the pose model. I changed the data path around it.

The application did more than pose detection. It decoded RTSP video, detected and tracked people, extracted torso regions for OCR, checked safety-related events, and generated an RTSP output stream.

The original path was roughly:

RTSP → OpenCV/FFmpeg → CPU frame
     → GPU inference → CPU-side processing
     → OCR → output

The TensorRT engine could execute on the GPU, but full frames still crossed the CPU/GPU boundary too often. Optimizing inference alone wasn't going to solve that.

A 1920×1080 RGB frame contains 6,220,800 bytes, or about 6.2 MB. One full-frame transfer at 25 FPS amounts to approximately 156 MB/s of image payload in one direction. This is a size calculation, not a measured PCIe bandwidth figure from the server.

Bandwidth wasn't the only concern. Intermediate buffers, format conversions, and synchronization can also add time. I needed to examine the frame's route through the application, not just the model's execution time.

The revised path was:

RTSP
  ↓
PyNvVideoCodec / NVDEC
  ↓
CUDA-resident frame
  ↓
GPU resize, letterbox, normalization
  ↓
Direct TensorRT inference
  ↓
GPU postprocessing and coordinate mapping
  ↓
Small metadata transfer for CPU tracking
  ↓
GPU torso ROI extraction
  ↓
OCR queue
  ↓
Downstream processing / RTSP output

I switched the decoding path to PyNvVideoCodec / NVDEC. This allowed decoded frames to remain in GPU-accessible memory instead of becoming CPU images that would then need to be uploaded for inference.

NVDEC doesn't make an entire application zero-copy by itself. The formats and buffers used by the next stages still matter. The useful change here was removing an unnecessary full-frame detour through host memory.

Resize, letterboxing, and normalization ran on the GPU. That let the pipeline prepare TensorRT input without routinely copying the full-resolution frame back to the CPU.

These operations aren't necessarily expensive individually. The problem is what happens when each stage creates another buffer or forces a transfer before the next stage can run.

I replaced the higher-level inference path with direct TensorRT execution using execute_async_v3().

That gave me explicit control over device buffers and execution. It wasn't the function name that made the pipeline faster: memory layout, buffer lifetime, and synchronization still had to be handled correctly. The benefit was being able to avoid work that the earlier path performed around inference.

The pose model returned person bounding boxes and 17 body keypoints. Postprocessing and coordinate mapping stayed on the GPU.

Tracking still used CPU-side information, but only compact detection metadata needed to cross to the CPU. Sending an entire 1080p frame just to update tracking coordinates would have defeated much of the optimization.

The pose keypoints defined torso regions for OCR. I extracted those ROIs on the GPU rather than copying the complete image to the CPU just to crop it.

Those ROIs then entered the OCR queue. This boundary matters when interpreting the performance number below.

I didn't try to move every part of the application to CUDA. I kept large pixel buffers on the GPU and moved smaller data when a CPU-side stage actually needed it.

The cover image is a frame from the tested pipeline, with tracking IDs and FPS counters overlaid.

The overlay showed:

PROCESSED: 24.64
OUTPUT:    25.00
SOURCE:    25

Compared with the original 8–10 FPS range, 24.64 FPS is about 2.5–3.1× the initial processing throughput.

The distinction between counters is important. The processed-FPS figure was measured up to the OCR queue. The 25 FPS output counter doesn't establish that OCR, safety checks, encoding, and network delivery all finished within 40 ms. Throughput and end-to-end latency are different measurements.

I also tested this architecture with four simultaneous camera streams on the same RTX 5070 Ti. I'm not publishing per-stream throughput or latency figures for that configuration here, so the 24.64 FPS result should be understood as the single-stream measurement.

If I encountered another RTSP application that wasn't keeping up with the camera, I'd check these boundaries before changing the model:

I'd also record stage timings and queue depth instead of relying on one FPS counter. A pipeline can appear fast at the inference stage while its end-to-end latency continues to grow.

For API details, NVIDIA documents PyNvVideoCodec decoding and interoperability and TensorRT asynchronous Python inference. These are references for implementing similar techniques, not code taken from the customer's deployment.

I'd diagram the entire frame path before spending time tuning inference: every buffer allocation, every CPU-to-GPU or GPU-to-CPU transfer, and every synchronization point.

In this project, that was where the meaningful improvement came from. A fast TensorRT engine didn't help much while the rest of the pipeline kept moving full-size frames around.

I shared more context and the original benchmark in my LinkedIn post about this RTSP/TensorRT optimization. If you've run into a similar bottleneck, I'd be interested to hear whether it was decoding, preprocessing, memory movement, or something else.

Implementation note: I configured and tested this pipeline on a customer's server. The performance figures come from that work. AI assistance was used to draft and edit the English text.