Unified Geometry-Instance 4D Reconstruction
IGGT4D unifies camera motion, scene geometry, and consistent instance features within a single feed-forward 4D reconstruction framework.
The paper, code, checkpoints, dataset, and benchmark will be released soon.
Real-world spatial intelligence requires agents to understand scenes from continuous video streams, where objects move, persist, disappear, and reappear over time. While recent spatial foundation models have enabled generalizable feed-forward 3D reconstruction, most streaming methods remain geometry-centric and lack temporally consistent object-level understanding. Meanwhile, existing semantic reconstruction and 3D-aware vision-language methods largely rely on externally extracted 2D semantic cues or loosely coupled geometry inputs, limiting unified geometry-instance learning in long dynamic scenes. In this paper, we propose IGGT4D, a streaming instance-grounded geometry Transformer for online 4D scene understanding. IGGT4D processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation of camera motion, geometry, and object identity. This enables long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, we further construct InsScene4D-147K, a large-scale dataset spanning real/synthetic and static/dynamic scenes, with RGB images, depth, poses, and temporally consistent instance masks generated by an automated geometry-guided annotation pipeline. Experiments on 3D reconstruction, pose estimation, instance spatial tracking, and open-vocabulary segmentation demonstrate that IGGT4D outperforms existing streaming baselines while maintaining scalable online inference for long dynamic sequences.
IGGT4D unifies camera motion, scene geometry, and consistent instance features within a single feed-forward 4D reconstruction framework.
Frame-by-frame causal inference and lightweight online clustering enable efficient reconstruction and persistent object identities over long 4D sequences.
We curate 147K real/synthetic and static/dynamic sequences, with 4D-consistent instance annotations from an automated geometry-guided pipeline.
Beyond existing public datasets, we further curate three complementary data sources for InsScene4D-147K:
Using the official RoboTwin 2.0 pipeline, we generate a new collection of RGB-D sequences with temporally consistent instance masks.
Using DA3 depth and ground-truth poses, our pipeline reconstructs a TSDF mesh for ID inheritance, then fuses SAM2 auto and box-prompted masks to refine annotations.
Our pipeline removes dynamic regions before static-mesh reconstruction, then updates projected pseudo-labels using provided dynamic-object annotations.
Causal attention and KV caching reuse historical context to build a unified geometry-instance representation frame by frame over long video streams.
The Tri-DPT head jointly decodes depth, rays, and instance features, using geometry-aware attention to inject geometric priors into instance representations.
Per-view cosine-similarity clustering groups instance features into instance masks, matches them to persistent IDs in a global codebook, and updates centers for streaming association.
Frame-by-frame visualization of RGB inputs, depth predictions, PCA-projected instance features, and instance masks in streaming video.
IGGT4D produces 4D-consistent instance features and masks that serve as reusable object-level cues for diverse downstream tasks, including spatial tracking, open-vocabulary segmentation, and LMM-based 4D scene grounding.
Streaming clustering assigns each local mask a persistent global ID. These frame-to-frame correspondences enable dense spatial tracking without an additional association model.
VLMs such as OpenSeg and LSeg extract per-frame semantics, while instance masks pool them into spatially and temporally consistent object features for open-vocabulary prediction.
4D-consistent instance features and masks provide cues to LMMs such as Qwen3-VL, enabling dynamic-object tracking, grounding, and spatial-temporal question answering.
Six quantitative tables report geometry reconstruction (camera pose estimation and 3D reconstruction using F1-score and Chamfer Distance), instance spatial tracking, open-vocabulary semantic segmentation, and streaming clustering efficiency.