Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

LidarVerse, a unified LiDAR world model presented in arXiv paper 2610.19201v1, achieves state-of-the-art performance on most reported metrics across all three evaluation protocols while running real-time inference at 11 FPS on a single H100 GPU. The model combines reconstruction-based and generative LiDAR simulation in one conditional generative framework, using a universal tokenizer that supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern. A four-step autoregressive causal student distilled from a bidirectional teacher, with rolling temporal K/V and world-aligned spatial caches, enables persistent long-horizon rollouts without per-scene optimization.

arXiv:2610.19201v1 Announce Type: new Abstract: We present LidarVerse, a real-time unified LiDAR world model for sensor simulation across tasks and sensors. Existing LiDAR simulators largely follow one of two complementary paradigms: reconstruction-based methods preserve the geometry of a captured scene but require lengthy per-scene optimization, whereas generative methods amortize learning across data but lack the consistency needed for long-horizon rollouts. LidarVerse brings both paradigms into a single, unified conditional generative framework. A universal tokenizer maps LiDAR range sequences into a shared latent space, where a bidirectional teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text. Varying the available scene evidence turns the same model into a generator of new worlds, a point-cloud forecaster, or a generalizable resimulator of observed scenes without per-scene optimization. We distill the teacher into a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon rollouts with real-time inference at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across all three evaluation protocols, while its tokenizer supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern. These results demonstrate that generative and reconstruction-grounded LiDAR simulation can be realized within one model, providing the core capabilities for future closed-loop sensor simulation.