{"slug": "iggt4d-streaming-4d-instance-grounded-geometry-transformer", "title": "IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer", "summary": "Researchers propose IGGT4D, a streaming instance-grounded geometry Transformer for online 4D scene understanding that unifies camera motion, scene geometry, and consistent instance features within a single feed-forward framework. The model processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation, enabling long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, the team constructed InsScene4D-147K, a large-scale dataset of 147,000 real/synthetic and static/dynamic sequences with temporally consistent instance annotations generated by an automated geometry-guided pipeline.", "body_md": "### Unified Geometry-Instance 4D Reconstruction\n\nIGGT4D unifies camera motion, scene geometry, and consistent instance features within a single feed-forward 4D reconstruction framework.\n\nThe paper, code, checkpoints, dataset, and benchmark will be released soon.\n\nReal-world spatial intelligence requires agents to understand scenes from continuous video streams, where objects move, persist, disappear, and reappear over time. While recent spatial foundation models have enabled generalizable feed-forward 3D reconstruction, most streaming methods remain geometry-centric and lack temporally consistent object-level understanding. Meanwhile, existing semantic reconstruction and 3D-aware vision-language methods largely rely on externally extracted 2D semantic cues or loosely coupled geometry inputs, limiting unified geometry-instance learning in long dynamic scenes. In this paper, we propose **IGGT4D**, a streaming instance-grounded geometry Transformer for online 4D scene understanding. IGGT4D processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation of camera motion, geometry, and object identity. This enables long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, we further construct **InsScene4D-147K**, a large-scale dataset spanning real/synthetic and static/dynamic scenes, with RGB images, depth, poses, and temporally consistent instance masks generated by an automated geometry-guided annotation pipeline. Experiments on 3D reconstruction, pose estimation, instance spatial tracking, and open-vocabulary segmentation demonstrate that IGGT4D outperforms existing streaming baselines while maintaining scalable online inference for long dynamic sequences.\n\nIGGT4D unifies camera motion, scene geometry, and consistent instance features within a single feed-forward 4D reconstruction framework.\n\nFrame-by-frame causal inference and lightweight online clustering enable efficient reconstruction and persistent object identities over long 4D sequences.\n\nWe curate 147K real/synthetic and static/dynamic sequences, with 4D-consistent instance annotations from an automated geometry-guided pipeline.\n\nBeyond existing public datasets, we further curate three complementary data sources for InsScene4D-147K:\n\nUsing the official RoboTwin 2.0 pipeline, we generate a new collection of RGB-D sequences with temporally consistent instance masks.\n\nUsing DA3 depth and ground-truth poses, our pipeline reconstructs a TSDF mesh for ID inheritance, then fuses SAM2 auto and box-prompted masks to refine annotations.\n\nOur pipeline removes dynamic regions before static-mesh reconstruction, then updates projected pseudo-labels using provided dynamic-object annotations.\n\nCausal attention and KV caching reuse historical context to build a unified geometry-instance representation frame by frame over long video streams.\n\nThe Tri-DPT head jointly decodes depth, rays, and instance features, using geometry-aware attention to inject geometric priors into instance representations.\n\nPer-view cosine-similarity clustering groups instance features into instance masks, matches them to persistent IDs in a global codebook, and updates centers for streaming association.\n\nFrame-by-frame visualization of RGB inputs, depth predictions, PCA-projected instance features, and instance masks in streaming video.\n\nIGGT4D produces 4D-consistent instance features and masks that serve as reusable object-level cues for diverse downstream tasks, including spatial tracking, open-vocabulary segmentation, and LMM-based 4D scene grounding.\n\nStreaming clustering assigns each local mask a persistent global ID. These frame-to-frame correspondences enable dense spatial tracking without an additional association model.\n\nVLMs such as OpenSeg and LSeg extract per-frame semantics, while instance masks pool them into spatially and temporally consistent object features for open-vocabulary prediction.\n\n4D-consistent instance features and masks provide cues to LMMs such as Qwen3-VL, enabling dynamic-object tracking, grounding, and spatial-temporal question answering.\n\nSix quantitative tables report geometry reconstruction (camera pose estimation and 3D reconstruction using F1-score and Chamfer Distance), instance spatial tracking, open-vocabulary semantic segmentation, and streaming clustering efficiency.", "url": "https://wpnews.pro/news/iggt4d-streaming-4d-instance-grounded-geometry-transformer", "canonical_source": "https://iggt4d.github.io/", "published_at": "2026-07-22 03:51:47+00:00", "updated_at": "2026-07-22 04:22:24.598457+00:00", "lang": "en", "topics": ["computer-vision", "machine-learning", "neural-networks", "artificial-intelligence"], "entities": ["IGGT4D", "InsScene4D-147K", "RoboTwin 2.0", "DA3", "SAM2", "OpenSeg", "LSeg"], "alternates": {"html": "https://wpnews.pro/news/iggt4d-streaming-4d-instance-grounded-geometry-transformer", "markdown": "https://wpnews.pro/news/iggt4d-streaming-4d-instance-grounded-geometry-transformer.md", "text": "https://wpnews.pro/news/iggt4d-streaming-4d-instance-grounded-geometry-transformer.txt", "jsonld": "https://wpnews.pro/news/iggt4d-streaming-4d-instance-grounded-geometry-transformer.jsonld"}}