{"slug": "i-benchmarked-9-object-detection-models-on-the-same-gpu-here-s-what-changed", "title": "I benchmarked 9 object-detection models on the same GPU — here's what changed", "summary": "A developer benchmarked nine object-detection models on a single GPU with a consistent measurement setup, finding that measured latencies were 1.6 to 11.6 times slower than published figures, significantly altering the speed ranking. The comparison also highlighted licensing differences across implementations, with the developer advising that architecture names alone are insufficient for commercial suitability assessments.", "body_md": "Published object-detection benchmarks are useful, but comparing them directly can be difficult.\n\nA model's published latency may depend on the GPU, precision, batch size, framework, TensorRT, and other optimizations used during testing. Two models with similar numbers on paper can behave very differently when run under the same conditions.\n\nI wanted to see what would happen if I removed some of those variables.\n\nSo I built a controlled comparison of **9 object-detection models**, using **one GPU, one measurement approach, and 48 scenes**.\n\nThe results changed the ranking considerably — and the licensing differences turned out to be just as interesting.\n\nThis is **not an accuracy benchmark**. The scenes currently do not have human-verified ground-truth annotations, so the comparison focuses on **inference latency, visual detections, confidence scores, and licensing**.\n\nHere are the published latency figures compared with the measurements from this setup:\n\n| Model | Published (ms) | Measured (ms) | Difference | \n|---|---|---|---|\n| YOLO11n | 1.5 | 17.4 | 11.6× | \n| RF-DETR-B | 6.0 | 69.1 | 11.5× | \n| RT-DETR-L | 9.3 | 66.2 | 7.1× | \n| YOLO26m | 4.2 | 20.6 | 4.9× | \n| YOLO11m | 5.6 | 20.8 | 3.7× | \n| YOLOv8m | 4.8 | 16.2 | 3.4× | \n| YOLOv9m | 7.3 | 23.9 | 3.3× | \n| Mask R-CNN R50 | 55.0 | 107.9 | 2.0× | \n| Faster R-CNN R50 | 45.0 | 73.3 | 1.6× | \n\nEvery model was slower in this setup than its published latency figure, with differences ranging from **1.6× to 11.6×**.\n\nThat doesn't necessarily mean the published benchmarks are wrong. They may use different hardware, precision, input sizes, optimization techniques, or measurement methods.\n\nThe important point is that **the numbers are not directly comparable unless the testing conditions are also comparable**.\n\nThe difference wasn't only about absolute latency.\n\nUsing the published numbers, the fastest-to-slowest ranking was:\n\nUnder the same measurement conditions, the ranking became:\n\nSome models moved significantly once they were measured under the same conditions.\n\nThis is one reason I find side-by-side comparisons more useful than simply collecting benchmark numbers from different project pages.\n\nThe benchmark setup was:\n\nThe models were loaded through their respective ecosystems, including Ultralytics, torchvision, Hugging Face Transformers, and the RF-DETR package.\n\nThe goal wasn't to create a universal benchmark. It was simply to answer a practical question:\n\n**What happens when these models are run through a consistent measurement setup?**\n\nThe comparison currently includes:\n\nThe test set contains **48 scenes** with different object types and conditions, including ordinary scenes as well as more challenging conditions such as fog, motion blur, aerial viewpoints, low light, snow, thermal imagery, and glare.\n\nThe purpose of these scenes is not to calculate mAP. Instead, they make it possible to visually inspect how different models behave when looking at exactly the same image.\n\nThe other part of the comparison that surprised me was licensing.\n\nFor a commercial project, the model's licence can sometimes be just as important as its speed.\n\nThe current models include licences such as:\n\nOne particularly interesting example is RT-DETR.\n\nThe licence can depend on **where and how the model is distributed**. The original Baidu release is available under Apache-2.0, while other implementations or packages may have different licensing terms.\n\nSo the architecture name alone is not always enough when evaluating whether a model is suitable for commercial use.\n\nThe practical lesson for me was simple:\n\n**Check the licence of the specific implementation, package, and weights you plan to use — not just the name of the architecture.**\n\nAfter comparing the models, three things stood out:\n\nDifferent hardware and optimization settings can change the result dramatically.\n\nA model that looks faster on a published benchmark may not remain faster when measured using a different but consistent setup.\n\nThe fastest or most accurate model isn't always the easiest one to use in a commercial product.\n\nI built an interactive version of the comparison where you can select the same scene and inspect how different models perform side by side.\n\nIt currently includes:\n\nYou can try it at:\n\nThere is a free demo available without signup.\n\nI'm planning to add **segmentation and pose estimation** next.\n\nIf you've had to choose an object-detection model for a real project, I'd be interested to know:\n\n**What mattered most in your decision — accuracy, latency, licence, hardware requirements, or something else?**", "url": "https://wpnews.pro/news/i-benchmarked-9-object-detection-models-on-the-same-gpu-here-s-what-changed", "canonical_source": "https://dev.to/cv-user203040/i-benchmarked-9-object-detection-models-on-the-same-gpu-heres-what-changed-2ppm", "published_at": "2026-09-08 07:48:57+00:00", "updated_at": "2026-09-08 08:01:35.810221+00:00", "lang": "en", "topics": ["computer-vision", "machine-learning", "developer-tools"], "entities": ["YOLO", "RF-DETR", "RT-DETR", "Mask R-CNN", "Faster R-CNN", "Ultralytics", "Hugging Face", "Baidu"], "alternates": {"html": "https://wpnews.pro/news/i-benchmarked-9-object-detection-models-on-the-same-gpu-here-s-what-changed", "markdown": "https://wpnews.pro/news/i-benchmarked-9-object-detection-models-on-the-same-gpu-here-s-what-changed.md", "text": "https://wpnews.pro/news/i-benchmarked-9-object-detection-models-on-the-same-gpu-here-s-what-changed.txt", "jsonld": "https://wpnews.pro/news/i-benchmarked-9-object-detection-models-on-the-same-gpu-here-s-what-changed.jsonld"}}