Beyond Gaussian Worlds: Latent Geometry Matters for JEPAs

A new arXiv paper (2609.21656v1) extends the analysis of Joint-Embedding Predictive Architectures (JEPAs) beyond Euclidean assumptions, showing that Gaussian uniqueness is not a universal property of distribution-matched JEPAs. The authors derive conditions under which alignment and exact distribution matching guarantee linear recovery for latent variables on embedded Riemannian manifolds, proving that uniformly spherical latents matched to a spherical target recover the latent state up to an orthogonal transformation. Experiments on Gaussian, spherical, and toroidal latent spaces show geometrically compatible targets yield better linear recovery, with the advantage persisting in high-dimensional Clifford-torus worlds.

arXiv:2609.21656v1 Announce Type: new Abstract: Recent Joint-Embedding Predictive Architectures JEPAs prevent representation collapse by constraining learned representations to follow a prescribed target distribution, such as an isotropic Gaussian or the uniform distribution on a hypersphere. Klindt et al. 2026 showed that, under their Euclidean assumptions, matching a Gaussian target can recover Gaussian latent variables up to a linear transformation, and that the Gaussian is the unique distribution with this guarantee. We extend their analysis to latent variables supported on embedded Riemannian manifolds and derive conditions on the latent geometry and positive-pair dynamics under which alignment and exact distribution matching guarantee linear recovery. In particular, when the latent variables are uniformly distributed on a sphere and the representations are matched to the same spherical distribution, every optimal representation recovers the latent state up to an orthogonal transformation. This shows that Gaussian uniqueness is not a universal property of distribution-matched JEPAs: non-Euclidean latent geometries can admit other linearly recoverable distributions. We further derive an approximate-recovery bound that is strictly tighter for the spherical world than for the Gaussian world. Experiments on Gaussian, spherical, and toroidal latent spaces show that geometrically compatible targets yield better linear recovery when optimization succeeds, whereas mismatched targets distort the latent structure. This advantage persists in high-dimensional Clifford-torus worlds.