Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

A study from the AI City Challenge 2026 Track 1 finds that a geometry-first RGB-only pipeline achieves 13.0 3D HOTA (51.6 LocA) in multi-camera 3D tracking under Sim2Real, while a pseudo-LiDAR approach using monocular depth collapses to 0.12 HOTA (9.2 LocA). The authors attribute the collapse to cross-view inconsistency of monocular depth, which domain-adaptation fine-tuning does not repair within budget. The geometry pipeline's bottleneck is detection quality (DetA), while pseudo-LiDAR is limited by localization consistency (LocA).

arXiv:2608.07579v1 Announce Type: new Abstract: The AI City Challenge 2026 Track 1 evaluates multi-camera 3D perception in large indoor warehouses under a synthetic-to-real Sim2Real setting; depth is available only for training and validation, so inference is RGB-only. We use two RGB-only routes as a controlled test of one hypothesis: that cross-view geometric consistency, not monocular depth accuracy, governs performance under Sim2Real. The first is a geometry-first pipeline: YOLO11x detection, homography lifting to the world frame, class-level 3D size priors, multi-camera fusion, world-coordinate tracking, and offline tracklet stitching. The second is estimated-depth pseudo-LiDAR: monocular depth D4RT, Metric3D~v2 back-projected into a fused point cloud and passed to a 3D detector V-DETR , mirroring prior point-cloud winners that used depth. The gap is decisive: geometry-first reaches 13.0 3D HOTA 51.6 LocA , whereas pseudo-LiDAR collapses to 0.12 9.2 LocA . We trace the collapse to cross-view inconsistency of monocular depth---scale correction is necessary but not sufficient---which domain-adaptation fine-tuning does not repair within budget. Within the geometry pipeline, offline stitching is the only intervention that helps; SAHI detection, appearance Re-ID, learned lifting, RT-DETR ensembling, test-time augmentation, and domain randomization all fail to beat the baseline detector. The bottlenecks are complementary: detection quality bounds the geometry route DetA , localization consistency bounds pseudo-LiDAR LocA . We release a complete, reproducible RGB-only pipeline and ablation.