IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer

Researchers propose IGGT4D, a streaming instance-grounded geometry Transformer for online 4D scene understanding that unifies camera motion, scene geometry, and consistent instance features within a single feed-forward framework. The model processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation, enabling long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, the team constructed InsScene4D-147K, a large-scale dataset of 147,000 real/synthetic and static/dynamic sequences with temporally consistent instance annotations generated by an automated geometry-guided pipeline.

Unified Geometry-Instance 4D Reconstruction IGGT4D unifies camera motion, scene geometry, and consistent instance features within a single feed-forward 4D reconstruction framework. The paper, code, checkpoints, dataset, and benchmark will be released soon. Real-world spatial intelligence requires agents to understand scenes from continuous video streams, where objects move, persist, disappear, and reappear over time. While recent spatial foundation models have enabled generalizable feed-forward 3D reconstruction, most streaming methods remain geometry-centric and lack temporally consistent object-level understanding. Meanwhile, existing semantic reconstruction and 3D-aware vision-language methods largely rely on externally extracted 2D semantic cues or loosely coupled geometry inputs, limiting unified geometry-instance learning in long dynamic scenes. In this paper, we propose IGGT4D , a streaming instance-grounded geometry Transformer for online 4D scene understanding. IGGT4D processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation of camera motion, geometry, and object identity. This enables long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, we further construct InsScene4D-147K , a large-scale dataset spanning real/synthetic and static/dynamic scenes, with RGB images, depth, poses, and temporally consistent instance masks generated by an automated geometry-guided annotation pipeline. Experiments on 3D reconstruction, pose estimation, instance spatial tracking, and open-vocabulary segmentation demonstrate that IGGT4D outperforms existing streaming baselines while maintaining scalable online inference for long dynamic sequences. IGGT4D unifies camera motion, scene geometry, and consistent instance features within a single feed-forward 4D reconstruction framework. Frame-by-frame causal inference and lightweight online clustering enable efficient reconstruction and persistent object identities over long 4D sequences. We curate 147K real/synthetic and static/dynamic sequences, with 4D-consistent instance annotations from an automated geometry-guided pipeline. Beyond existing public datasets, we further curate three complementary data sources for InsScene4D-147K: Using the official RoboTwin 2.0 pipeline, we generate a new collection of RGB-D sequences with temporally consistent instance masks. Using DA3 depth and ground-truth poses, our pipeline reconstructs a TSDF mesh for ID inheritance, then fuses SAM2 auto and box-prompted masks to refine annotations. Our pipeline removes dynamic regions before static-mesh reconstruction, then updates projected pseudo-labels using provided dynamic-object annotations. Causal attention and KV caching reuse historical context to build a unified geometry-instance representation frame by frame over long video streams. The Tri-DPT head jointly decodes depth, rays, and instance features, using geometry-aware attention to inject geometric priors into instance representations. Per-view cosine-similarity clustering groups instance features into instance masks, matches them to persistent IDs in a global codebook, and updates centers for streaming association. Frame-by-frame visualization of RGB inputs, depth predictions, PCA-projected instance features, and instance masks in streaming video. IGGT4D produces 4D-consistent instance features and masks that serve as reusable object-level cues for diverse downstream tasks, including spatial tracking, open-vocabulary segmentation, and LMM-based 4D scene grounding. Streaming clustering assigns each local mask a persistent global ID. These frame-to-frame correspondences enable dense spatial tracking without an additional association model. VLMs such as OpenSeg and LSeg extract per-frame semantics, while instance masks pool them into spatially and temporally consistent object features for open-vocabulary prediction. 4D-consistent instance features and masks provide cues to LMMs such as Qwen3-VL, enabling dynamic-object tracking, grounding, and spatial-temporal question answering. Six quantitative tables report geometry reconstruction camera pose estimation and 3D reconstruction using F1-score and Chamfer Distance , instance spatial tracking, open-vocabulary semantic segmentation, and streaming clustering efficiency.