Lingbot-map: A 3D foundation model for reconstructing scenes from streaming data

LingBot-Map, a feed-forward 3D foundation model for streaming scene reconstruction, achieves state-of-the-art performance at ~20 FPS on long sequences exceeding 10,000 frames. The model uses a Geometric Context Transformer with paged KV cache attention and has been released with evaluation benchmarks and demo scripts for multiple datasets.

teaser.mp4 🗺️ Meet LingBot-Map We've built a feed-forward 3D foundation model for streaming 3D reconstruction 🏗️🌍 LingBot-Map has focused on: Geometric Context Transformer : Architecturally unifies coordinate grounding, dense geometric cues, and long-range drift correction within a single streaming framework through anchor context, pose-reference window, and trajectory memory. High-Efficiency Streaming Inference : A feed-forward architecture with paged KV cache attention, enabling stable inference at ~20 FPS on 518×378 resolution over long sequences exceeding 10,000 frames. State-of-the-Art Reconstruction : Superior performance on diverse benchmarks compared to both existing streaming and iterative optimization-based approaches. Click to expand 2026-06-28 — Fixed an SDPA KV cache bug. The SDPA backend now performs better on long sequences . We still recommend the FlashInfer backend for the best performance. 2026-05-25 — 📊 Evaluation benchmark released . We released the evaluation scripts for KITTI and Oxford Spires — see benchmark/ /Robbyant/lingbot-map/blob/main/benchmark for the pipeline, and runto prepare Oxford Spires data before evaluation. preprocess/oxford.py 2026-04-29 — 📹 Long-video demo released . We released a very-long-video example ~25 000 frames, 13-minute indoor walkthrough rendered with the offline pipeline — see Worked Example worked-example--long-indoor-walkthrough-25-000-frames-13-minutes for the command, flag rationale, and rendered output. 2026-04-27 — 🚀 LingBot-Map accelerated . Pull the latest main and run python demo.py --compile ... or python gct profile.py --backend flashinfer --dtype bf16 --compile to verify on your hardware. 2026-04-24 — Fixed a FlashInfer KV cache bug where --keyframe interval 1 silently cached non-keyframes. You should now see better pose and reconstruction quality when running with more than 320 frames . - ✅ Release evaluation benchmark - ✅ Oxford Spires dataset - ✅ KITTI dataset - ✅ VBR dataset - ✅ Droid-W dataset - ✅ TUM-D dataset - ✅ 7-scenes dataset - ✅ ETH3D dataset - ✅ Tanks and Temples dataset - ✅ NRGBD dataset - ✅ Release demo scripts - ✅ Indoor long-video demo Featured indoor walkthrough -featured-indoor-walkthrough-25-000-frames-13-minutes - ✅ Outdoor long-video demo - ✅ LingBot-World demo Worked example worked-example--lingbot-world-scenes - ✅ Aerial long-video demo - ✅ Indoor long-video demo 1. Create conda environment conda create -n lingbot-map python=3.10 -y conda activate lingbot-map 2. Install PyTorch CUDA 12.8 pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128 PyTorch 2.8.0 is the recommended version because NVIDIA Kaolin required by the batch rendering pipeline has prebuilt wheels for torch-2.8.0 cu128 . If you only need demo.py you may use a newer PyTorch, but the batch renderer then requires building Kaolin from source. For other CUDA versions, see PyTorch Get Started . 3. Install lingbot-map pip install -e . 4. Install FlashInfer recommended FlashInfer provides paged KV cache attention for efficient streaming inference. It is a pure-Python package that JIT-compiles CUDA kernels on first use, so a single wheel works across CUDA/PyTorch versions: pip install --index-url https://pypi.org/simple flashinfer-python --index-url https://pypi.org/simple is only needed if your default pip index is an internal mirror that doesn't have flashinfer-python . Optional For faster first-use, you can additionally install a CUDA-specific JIT cache: pip install flashinfer-jit-cache -f https://flashinfer.ai/whl/cu128/flashinfer-jit-cache/ . See FlashInfer installation for details. If FlashInfer is not installed, the model falls back to SDPA PyTorch native attention via --use sdpa . 5. Visualization dependencies optional pip install -e ". vis " | Model Name | Huggingface Repository | ModelScope Repository | Description | |---|---|---|---| | lingbot-map-long | | Robbyant/lingbot-map https://www.modelscope.cn/models/Robbyant/lingbot-map robbyant/lingbot-map https://huggingface.co/robbyant/lingbot-map Robbyant/lingbot-map https://www.modelscope.cn/models/Robbyant/lingbot-map robbyant/lingbot-map https://huggingface.co/robbyant/lingbot-map Robbyant/lingbot-map https://www.modelscope.cn/models/Robbyant/lingbot-map 🚧 Coming soon:we're training an stronger model that supports longer sequences — stay tuned. After installation, run your first scene with one command: python demo.py --model path /path/to/lingbot-map-long.pt \ --image folder example/courthouse --mask sky This launches an interactive viser https://github.com/nerfstudio-project/viser viewer at http://localhost:8080 . See Interactive Demo -interactive-demo-demopy below for the full set of scenes and flags, or jump to Offline Rendering Pipeline -offline-rendering-pipeline-demo renderbatch demopy for long-sequence batch rendering. Run demo.py for interactive 3D visualization via a browser-based viser https://github.com/nerfstudio-project/viser viewer default http://localhost:8080 . We provide four example scenes in example/ that you can run out of the box: courthouse scene python demo.py --model path /path/to/lingbot-map-long.pt \ --image folder example/courthouse --mask sky output pointcloud side by side.mp4 University scene python demo.py --model path /path/to/lingbot-map-long.pt \ --image folder example/university --mask sky output pointcloud side by side.mp4 Loop scene loop closure trajectory python demo.py --model path /path/to/lingbot-map-long.pt \ --image folder example/loop output pointcloud side by side.mp4 Oxford scene with sky masking outdoor, large scale scene python demo.py --model path /path/to/lingbot-map-long.pt \ --image folder example/oxford --mask sky output pointcloud side by side.mp4 Sequence is too long for the interactive viser viewer — this clip was rendered with the Offline Rendering Pipeline. See that section for the full command. We will provide more examples in the follow-up. Use --keyframe interval to reduce KV cache memory by only keeping every N-th frame as a keyframe. Non-keyframe frames still produce predictions but are not stored in the cache. This is useful for long sequences which exceed 320 frames We train with video RoPE on 320 views, so performance degrades when the KV cache stores more than 320 views. Using a keyframe strategy allows inference over longer sequences. . Dataset: Download the demo sequences from robbyant/lingbot-map-demo https://huggingface.co/datasets/robbyant/lingbot-map-demo/tree/main on Hugging Face. Example run on the travel sequence from the dataset above sky masking on, 4 camera optimization iterations, keyframe every 2 frames : python demo.py \ --image folder /path/to/lingbot-map-demo/travel/ \ --model path /path/to/lingbot-map-long.pt \ --mask sky \ --camera num iterations 4 \ --keyframe interval 2 output pointcloud side by side.mp4 Note on inference range.Our method does not perform state resetting by default, so the maximum inference range is bounded by the longest distance seen during training on the dataset. Beyond that distance, state resetting becomes necessary. If you observe pose collapse, switch to windowed mode --mode windowed — in most cases tuning --keyframe interval alone is enough and the rest of the windowed parameters can stay at their defaults. python demo.py --model path /path/to/lingbot-map-long.pt \ --video path video.mp4 --fps 10 \ --mode windowed --window size 128 --overlap keyframes 16 --keyframe interval 2 Sky masking uses an ONNX sky segmentation model to filter out sky points from the reconstructed point cloud, which improves visualization quality for outdoor scenes. Setup: Install onnxruntime required pip install onnxruntime CPU or pip install onnxruntime-gpu GPU faster for large image sets The sky segmentation model skyseg.onnx will be automatically downloaded from HuggingFace https://huggingface.co/JianyuanWang/skyseg/resolve/main/skyseg.onnx on first use. Usage: python demo.py --model path /path/to/checkpoint.pt \ --image folder /path/to/images/ --mask sky Sky masks are cached in <image folder sky masks/ so subsequent runs skip regeneration. You can also specify a custom cache directory with --sky mask dir , or save side-by-side mask visualizations with --sky mask visualization dir : python demo.py --model path /path/to/checkpoint.pt \ --image folder /path/to/images/ --mask sky \ --sky mask dir /path/to/cached masks/ \ --sky mask visualization dir /path/to/mask viz/ | Argument | Default | Description | |---|---|---| --port | 8080 | Viser viewer port | --conf threshold | 1.5 | Visibility threshold for filtering low-confidence points | --point size | 0.00001 | Point cloud point size | --downsample factor | 10 | Spatial downsampling for point cloud display | python demo.py --model path /path/to/checkpoint.pt \ --image folder /path/to/images/ --use sdpa If you run into out-of-memory issues, try one or both of the following: — offload per-frame predictions to CPU during inference on by default; use --offload to cpu --no-offload to cpu only if you have memory to spare .— reduce the number of bidirectional scale frames from the default 8 down to 2, which shrinks the activation peak of the initial scale phase. --num scale frames 2 Lower the number of iterative refinement steps in the camera head to trade a small amount of pose accuracy for wall-clock speed: python demo.py --model path /path/to/checkpoint.pt \ --image folder /path/to/images/ --camera num iterations 1 --camera num iterations defaults to 4 ; setting it to 1 skips three refinement passes in the camera head and shrinks its KV cache by 4× . Use this pipeline when your sequence is too long for the interactive viser viewer — for example, the indoor walkthrough featured above -featured-indoor-walkthrough-25-000-frames-13-minutes . demo render/batch demo.py is the all-in-one offline entry point: feed it a video or a folder of images and it will run model inference and produce a headless point-cloud flythrough MP4 in a single command. It shares the same PyTorch / FlashInfer / checkpoint stack as demo.py . For those constrained by limited VRAM or GPU usage, you may also refer to the implementation at: https://github.com/ureeey/lingbot-map-rtx4060-8g/commit/eeee84a89cc97c1e39b736b46df4ee315275700b https://github.com/ureeey/lingbot-map-rtx4060-8g/commit/eeee84a89cc97c1e39b736b46df4ee315275700b 1. Rendering Python dependencies pip install -e ". vis,render " render pulls in open3d =0.19 and pyyaml the core numpy<2 constraint comes from the base lingbot-map install . Sky masking in this pipeline uses onnxruntime-gpu for batched segmentation; install it if you don't already have the CPU onnxruntime : pip install onnxruntime-gpu 2. Kaolin — matches the PyTorch 2.8.0 + CUDA 12.8 recommended above: pip install --index-url https://pypi.org/simple \ kaolin -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.8.0 cu128.html --index-url https://pypi.org/simple bypasses any internal mirror that might otherwise serve the PyPI placeholder wheel which raises ImportError on import . NVIDIA Kaolin does not publish prebuilt wheels for PyTorch 2.9.x — if you're on 2.9 for other reasons, build Kaolin from source pip install --no-build-isolation git+https://github.com/NVIDIAGameWorks/kaolin.git , needs local CUDA toolkit . For other torch/CUDA combinations see NVIDIA Kaolin installation . 3. ffmpeg sudo apt install ffmpeg or: brew install ffmpeg 4. CUDA extensions required before first run cd demo render/render cuda ext && python setup.py build ext --inplace && cd ../.. This builds voxel morton ext and frustum cull ext in place — both are imported by rgbd render for GPU voxelization and frustum culling. Dataset: Download the example video from robbyant/lingbot-map-demo https://huggingface.co/datasets/robbyant/lingbot-map-demo/tree/main on Hugging Face. python demo render/batch demo.py \ --video path /data/demo videos/indoor travel.MP4 \ --output folder /data/outputs/indoor travel/ \ --model path /path/to/lingbot-map.pt \ --config demo render/config/indoor.yaml \ --mode windowed --window size 128 \ --keyframe interval 10 --overlap keyframes 8 \ --sky mask dir /data/outputs/sky masks \ --sky mask visualization dir /data/outputs/sky mask viz \ --camera vis default --keyframes only points \ --frame tag --frame tag position top right \ --save predictions Flag-by-flag rationale: | Flag | Why it's there | |---|---| --mode windowed --window size 128 | Sliding-window inference is required once the sequence exceeds the ~320-frame RoPE training range; each window resets the KV cache. — the first window size counts KV-cache slots, not actual frames num scale frames =8 slots hold the scale frames and the remaining 128 − 8 = 120 slots hold keyframes. With keyframe interval = 13 , one window therefore covers 8 + 120 × 13 = 1568 actual frames. | --keyframe interval 10 | Cache only every 10th frame as a keyframe. Non-keyframes still emit per-frame predictions but don't grow the KV cache | --overlap keyframes 8 | Adjacent windows share 8 keyframes of context, resolved internally to max num scale frames, 8 × keyframe interval = 8 × 13 = 104 actual frames of overlap. Recommended whenever keyframe interval 1 , to keep cross-window pose alignment stable. | --config demo render/config/indoor.yaml | Seed render/scene/camera/overlay defaults from the indoor preset short depth, tighter follow cam . Any CLI flag the user explicitly passes still overrides the YAML value. | --sky mask dir / --sky mask visualization dir | Persist sky masks and their side-by-side visualizations to disk so subsequent reruns reuse them instead of re-running ONNX segmentation. The render pipeline only consumes them when sky masking is enabled — by the YAML preset or by --mask sky . | --camera vis default | Overlay the trajectory trail + recent-frame points on the rendered video. | --keyframes only points | Only unproject keyframe depth into the point cloud; non-keyframes still contribute their pose to the trajectory/frustum overlay. Keeps the cloud sparse for very long sequences. | --frame tag --frame tag position top right | Stamp a <i / <N Frames counter in the top-right corner of the MP4. | --save predictions | Persist per-frame NPZs alongside the MP4. Useful for inspection or for re-rendering with different camera/overlay settings later. | Replacing keyframe interval = 10 with image stride = 10 speeds up rendering. Then, comment out the camera follow section in demo render/config/indoor.yaml and set the birdeye's ranges to 2000, 2500 to reproduce the indoor fly-through effect shown in the demo: Dataset: Download the example video from robbyant/lingbot-map-demo https://huggingface.co/datasets/robbyant/lingbot-map-demo/tree/main on Hugging Face. python demo render/batch demo.py \ --video path /data/demo videos/drive frames.mp4 \ --output folder /data/outputs/drive/ \ --model path /path/to/lingbot-map.pt \ --config demo render/config/outdoor drive.yaml \ --mode windowed --window size 128 \ --max non keyframe gap 100 --overlap keyframes 8 \ --image stride 1 \ --sky mask dir /data/outputs/sky masks \ --sky mask visualization dir /data/outputs/sky mask viz \ --camera vis default --keyframes only points \ --frame tag --frame tag position top right \ --save predictions What differs from the indoor walkthrough above: | Flag | Why it's there | |---|---| --config demo render/config/outdoor drive.yaml | Seed defaults from the outdoor preset: sky masking enabled, deeper render range max depth: 250 , and a follow cam tuned for vehicle trajectories with a final birdeye reveal. | --image stride 1 | Use every video frame. Increase it to subsample long or high-FPS drive footage. | --max non keyframe gap 100 | Upper bound on consecutive non-keyframes before a keyframe is forced. Only active with flow-based keyframe selection --flow threshold 0 ; in the default fixed-interval mode it has no effect. | The remaining flags --mode windowed --window size 128 , --overlap keyframes 8 , sky-mask caching, overlays, --save predictions carry over unchanged from the indoor example — see the flag-by-flag table above. Reconstruct videos generated by LingBot-World, our world model — the same pipeline works on generated footage out of the box. Dataset: Download the example videos lingbo world frames.mp4 , lingbo world2 frames.mp4 from robbyant/lingbot-map-demo https://huggingface.co/datasets/robbyant/lingbot-map-demo/tree/main on Hugging Face. python demo render/batch demo.py \ --video path /data/demo videos/lingbo world frames.mp4 \ --output folder /data/outputs/lingbo world/ \ --model path /path/to/lingbot-map.pt \ --config demo render/config/outdoor drive.yaml \ --mode windowed --window size 128 \ --max non keyframe gap 100 --overlap keyframes 8 \ --image stride 1 \ --sky mask dir /data/outputs/sky masks \ --sky mask visualization dir /data/outputs/sky mask viz \ --camera vis default --keyframes only points \ --frame tag --frame tag position top right \ --save predictions For the second clip, run the same command with --video path /data/demo videos/lingbo world2 frames.mp4 --output folder /data/outputs/lingbo world2/ and separate --sky mask dir / --sky mask visualization dir folders if you want to keep the cached masks apart . All flags are identical to the outdoor drive scene worked-example--outdoor-drive-scene above — only the input video and output folder change. See the drive scene and indoor walkthrough tables for the flag-by-flag rationale. The virtual camera path is described by the camera.segments list in the YAML preset passed via --config . Edit the YAML to design your own shot — no need to touch CLI flags. Built-in presets live in demo render/config/ : default.yaml , indoor.yaml , outdoor drive.yaml . Copy one and edit the camera: block. camera: fov: 60.0 camera field of view in degrees transition: 30 frames blended between adjacent segments segments: - mode: follow chase cam following the input trajectory frames: 0, 1500 rendered-frame range this segment covers -1 = end back offset: 0.3 how far behind the input camera fraction of scene scale up offset: 0.08 vertical lift above the input camera look offset: 0.4 how far ahead the lookat target points smooth window: 30 trajectory smoothing window in frames - mode: birdeye rise up for a top-down reveal of the whole scene frames: 1500, 1800 reveal height mult: 2.5 birdeye height = scene scale × this factor - mode: follow drop back into chase cam frames: 1800, -1 back offset: 0.3 up offset: 0.08 look offset: 0.4 transition controls how many frames are blended between adjacent segments; frames: 0, -1 means "the whole sequence". mode | Behavior | Tunable fields | |---|---|---| follow | Chase cam tracks the input trajectory with smooth offsets. The most cinematic option for walkthroughs. | back offset , up offset , look offset , smooth window , scale frames | birdeye | Top-down reveal of the whole scene. Useful for hero / overview shots. | reveal height mult | static | Fixed eye + lookat, auto-derived from the segment's start frame. | — | pivot | Fixed eye, lookat sweeps along the trajectory. | — | Pure follow most common : camera: fov: 60.0 segments: - mode: follow frames: 0, -1 back offset: 0.3 up offset: 0.08 look offset: 0.4 smooth window: 30 Full birdeye good for overview / hero shots : camera: fov: 60.0 segments: - mode: birdeye frames: 0, -1 reveal height mult: 2.5 Follow with birdeye inserts : just list multiple segments in order under segments: — adjacent segments are interpolated using transition frames. Caveat: when --config loads a YAML preset, passinganysegment-shaping CLI flag --camera mode , --back offset , --up offset , --look offset , --smooth window , --follow scale frames , --birdeye start , --birdeye duration , --reveal height mult discards the YAML's segments and rebuilds the camera path from those flags instead. To stay fully YAML-driven, don't pass any of them on the command line. For a given output name e.g. <scene or <video name : | File | Description | |---|---| <name pointcloud.mp4 | Rendered point-cloud flythrough | <name pointcloud rgb.mp4 | Original RGB frames encoded as video | <name pointcloud config.yaml | Full config snapshot of this run | batch results.json | Per-scene success / duration summary | This project is released under the Apache License 2.0. See LICENSE /Robbyant/lingbot-map/blob/main/LICENSE.txt file for details. @article{chen2026geometric, title={Geometric Context Transformer for Streaming 3D Reconstruction}, author={Chen, Lin-Zhuo and Gao, Jian and Chen, Yihang and Cheng, Ka Leong and Sun, Yipengjing and Hu, Liangxiao and Xue, Nan and Zhu, Xing and Shen, Yujun and Yao, Yao and Xu, Yinghao}, journal={arXiv preprint arXiv:2604.14141}, year={2026} } We thank Shangzhan Zhang, Jianyuan Wang, Yudong Jin, Christian Rupprecht, and Xun Cao for their helpful discussions and support. This work builds upon several excellent open-source projects: