Decoupled Drift-Diffusion World Models for Consistent Future Frame Prediction

Researchers posted arXiv:2610.19212v1, describing Decoupled Drift-Diffusion World Model (D3WM), a transformer trained with a Gaussian negative-log-likelihood objective that jointly predicts the deterministic next frame and a spatial map of heteroscedastic aleatoric uncertainty, then uses that map to gate a conditional diffusion process. D3WM reports improved results over competitive baselines in two action spaces: robotic tabletop manipulation and camera motion. The method frames the decomposition as a variance-gated stochastic differential equation separating deterministic drift from stochastic diffusion.

arXiv:2610.19212v1 Announce Type: new Abstract: World models of visual information predict future image frames conditioned on given action sequences. This predicted scene evolution is driven by deterministic changes such as controlled motion on a robot platform, as well as stochastic changes such as rendering of manipulated objects or previously occluded regions. Our key insight is that the spatial location of these stochastic changes is itself learnable. We leverage this insight by jointly regressing both the deterministic next-state prediction and the spatial uncertainty map, then using the uncertainty map to guide a conditional diffusion process. We propose Decoupled Drift-Diffusion World Model D3WM , which uses a transformer trained with a Gaussian negative-log-likelihood objective to produce both the deterministic next-state prediction and a spatial map of heteroscedastic aleatoric uncertainty. The learned uncertainty map then provides a structured spatial gate for a conditional diffusion model: deterministic regions are anchored to the predicted state, while uncertain regions, such as disocclusions or dynamic textures, are sampled by the diffusion process. We frame this decomposition under a variance-gated stochastic differential equation that separates deterministic drift from stochastic diffusion. D3WM shows improved results over competitive baselines in two distinct types of action spaces: robotic tabletop manipulation and camera motion.