Published object-detection benchmarks are useful, but comparing them directly can be difficult.
A model's published latency may depend on the GPU, precision, batch size, framework, TensorRT, and other optimizations used during testing. Two models with similar numbers on paper can behave very differently when run under the same conditions.
I wanted to see what would happen if I removed some of those variables.
So I built a controlled comparison of 9 object-detection models, using one GPU, one measurement approach, and 48 scenes.
The results changed the ranking considerably — and the licensing differences turned out to be just as interesting.
This is not an accuracy benchmark. The scenes currently do not have human-verified ground-truth annotations, so the comparison focuses on inference latency, visual detections, confidence scores, and licensing.
Here are the published latency figures compared with the measurements from this setup:
| Model | Published (ms) | Measured (ms) | Difference |
|---|---|---|---|
| YOLO11n | 1.5 | 17.4 | 11.6× |
| RF-DETR-B | 6.0 | 69.1 | 11.5× |
| RT-DETR-L | 9.3 | 66.2 | 7.1× |
| YOLO26m | 4.2 | 20.6 | 4.9× | | YOLO11m | 5.6 | 20.8 | 3.7× | | YOLOv8m | 4.8 | 16.2 | 3.4× | | YOLOv9m | 7.3 | 23.9 | 3.3× | | Mask R-CNN R50 | 55.0 | 107.9 | 2.0× | | Faster R-CNN R50 | 45.0 | 73.3 | 1.6× |
Every model was slower in this setup than its published latency figure, with differences ranging from 1.6× to 11.6×.
That doesn't necessarily mean the published benchmarks are wrong. They may use different hardware, precision, input sizes, optimization techniques, or measurement methods.
The important point is that the numbers are not directly comparable unless the testing conditions are also comparable.
The difference wasn't only about absolute latency.
Using the published numbers, the fastest-to-slowest ranking was: Under the same measurement conditions, the ranking became:
Some models moved significantly once they were measured under the same conditions.
This is one reason I find side-by-side comparisons more useful than simply collecting benchmark numbers from different project pages.
The benchmark setup was:
The models were loaded through their respective ecosystems, including Ultralytics, torchvision, Hugging Face Transformers, and the RF-DETR package.
The goal wasn't to create a universal benchmark. It was simply to answer a practical question:
What happens when these models are run through a consistent measurement setup?
The comparison currently includes:
The test set contains 48 scenes with different object types and conditions, including ordinary scenes as well as more challenging conditions such as fog, motion blur, aerial viewpoints, low light, snow, thermal imagery, and glare.
The purpose of these scenes is not to calculate mAP. Instead, they make it possible to visually inspect how different models behave when looking at exactly the same image.
The other part of the comparison that surprised me was licensing.
For a commercial project, the model's licence can sometimes be just as important as its speed. The current models include licences such as:
One particularly interesting example is RT-DETR.
The licence can depend on where and how the model is distributed. The original Baidu release is available under Apache-2.0, while other implementations or packages may have different licensing terms.
So the architecture name alone is not always enough when evaluating whether a model is suitable for commercial use.
The practical lesson for me was simple:
Check the licence of the specific implementation, package, and weights you plan to use — not just the name of the architecture.
After comparing the models, three things stood out:
Different hardware and optimization settings can change the result dramatically.
A model that looks faster on a published benchmark may not remain faster when measured using a different but consistent setup.
The fastest or most accurate model isn't always the easiest one to use in a commercial product.
I built an interactive version of the comparison where you can select the same scene and inspect how different models perform side by side.
It currently includes:
You can try it at:
There is a free demo available without signup.
I'm planning to add segmentation and pose estimation next.
If you've had to choose an object-detection model for a real project, I'd be interested to know: What mattered most in your decision — accuracy, latency, licence, hardware requirements, or something else?