Interpreting Physics in Video Encoders using Multi-Object Interaction Videos

A new arXiv paper (2610.31983v1) probing relational physics attributes in two frozen self-supervised video encoders, V-JEPA2 and VideoMAE, found mixed results under two-body collision settings. Both encoders reliably separate elastic from inelastic collisions, but momentum-violation detection appears in only one of the two encoders, a kinetic-energy readout survives an environment change for V-JEPA2 but not VideoMAE, and neither model holds a friction coefficient that survives a change of geometry. The authors frame the work around video encoders used as world models for physical AI, where what the latent holds bounds what an agent can plan over.

arXiv:2610.31983v1 Announce Type: new Abstract: Video encoder models are increasingly used as world models for physical AI, where a policy plans by rolling out the latent, so what physics the latent holds bounds what the agent can plan over. Moreover, existing interpretability work probes quantities belonging to a single object; the quantities that make physics relational, such as momentum transfer, restitution, and friction, are defined between bodies and have not been examined. We probe relational physics attributes in two frozen self-supervised video encoders V-JEPA2 and VideoMAE under two-body settings and test if they transfer across environments. Our early findings are mixed. Both encoders separate elastic from inelastic collisions reliably. V-JEPA2's own predictor is less likely to predict a collision that creates kinetic energy compared to one that loses it. Momentum-violation detection appears in only one of the two encoders, a kinetic-energy readout survives a change of environment for V-JEPA2 but not for VideoMAE, and neither model holds a friction coefficient that survives a change of geometry.