Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency

Researchers introduced Action-Conditioned Predictive Consistency (ACPC), a diagnostic for Joint-embedding predictive architectures (JEPAs) that measures divergence between clean and visually perturbed histories under the same action sequence, proving it bounds perturbation-induced changes in prediction error and planner cost. Experiments on four visual control tasks showed pairwise ACPC predicts perturbation-induced changes, and the Invariance Radius (IR) and Separation Rate (SR) measures transfer across tasks on LeWM, remaining informative under blur and resize.

arXiv:2608.12939v1 Announce Type: new Abstract: Joint-embedding predictive architectures JEPAs learn world models that predict in a compact latent space rather than in pixels, reducing the pressure to model nuisance appearance. Yet this provides no guarantee against visual perturbations: they can still alter the encoded representation and affect subsequent action-conditioned predictions. Bisimulation captures this requirement precisely: two observations should be treated as the same state only when their action-conditioned consequences agree. Guided by this criterion, we introduce Action-Conditioned Predictive Consistency ACPC , a diagnostic that measures how far a clean history and a visually perturbed view of it diverge after being rolled forward under the same action sequence. We prove that this divergence bounds the perturbation-induced change in multi-step prediction error and planner cost. Building on pairwise ACPC, we define two complementary measures: the Invariance Radius IR summarizes clean-perturbed rollout spread, while the Separation Rate SR checks whether different states remain distinguishable after rollout. Experiments on four visual control tasks show that pairwise ACPC predicts perturbation-induced prediction and cost changes. On LeWM, the IR-SR screen transfers across tasks, and the joint diagnostic remains informative under blur and resize. PLDM exhibits similar diagnostic trends under a different architecture.