{"slug": "levjepa-efficient-and-scalable-video-pretraining-without-the-heuristics", "title": "LeVJEPA: Efficient and Scalable Video Pretraining Without the Heuristics", "summary": "LeVJEPA, a new video pretraining method from researchers at the German Cancer Research Center, German Cancer Consortium, Goethe University Frankfurt, Mila, Université de Montréal, Brown University, Courant Institute at New York University, and Advanced Machine Intelligence (AMI Labs), matches or exceeds V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8 times lower total pretraining compute, and at matched FLOPs exceeds the strongest video baseline by 7.6 points on ImageNet-1K. The method eliminates the need for a target encoder, stop-gradient, and predictor by using SIGReg regularization to prevent collapse, and it achieves higher accuracy with increased token dropping, rising from 33.9% at rho=0 to 47.6% at rho=0.95 on ImageNet-1K.", "body_md": "# LeVJEPA: Efficient & Scalable Video Pretraining\n\nwithout the Heuristics\n\n1German Cancer Research Center\n\n2German Cancer Consortium\n\n3Goethe University Frankfurt\n\n4Mila\n\n5Université de Montréal\n\n6Brown University\n\n7Courant Institute, New York University\n\n8Advanced Machine Intelligence (AMI Labs)\n\n*Equal advising. Correspondence:\n`lukas.kuhn@dkfz-heidelberg.de`\n\n## Abstract\n\nVideo provides an abundant, unannotated record of the temporal structure of the physical world, yet\nself-supervised learning from it has remained computationally costly. Prevailing joint-embedding methods\npreclude representation collapse through architectural asymmetry — an exponential-moving-average\ntarget encoder, a stop-gradient, and a capacity-limited predictor — while masked-autoencoding methods\ncircumvent the question by reconstructing pixels through a dedicated decoder. We introduce\n**LeVJEPA**, the first video encoder trained under the collapse-free objective of LeJEPA,\nwhich requires neither. A single encoder is optimized with an invariance loss over global and local views\nof a clip, regularized by SIGReg, which precludes collapse under a provable guarantee. The trainable\narchitecture reduces to an encoder and a projector, and the objective retains a single hyperparameter. At\nmatched epochs on identical data, LeVJEPA matches or exceeds V-JEPA 2 across ViT-S/B/L at \\(5.6\\) to\n\\(20.8\\times\\) lower total pretraining compute; at matched FLOPs it exceeds the strongest video baseline by\n\\(7.6\\) points on ImageNet-1K. Because no asymmetry between branches is required, the encoder admits\nblock-causal attention at no measurable cost in accuracy, such that each frame representation depends only\non past observations.\n\n## Objective and architecture\n\nFrom each video we sample a clip of \\(16\\) frames and construct \\(V+1\\) views: one global view at full\nresolution and \\(V\\) local views obtained by spatial cropping and photometric augmentation, all sharing an\nidentical temporal window. Every view is processed by the same encoder \\(E_\\theta\\); a learnable\n`[cls]`\n\ntoken provides the clip-level readout, which a projector maps to an embedding\n\\(z_v \\in \\mathbb{R}^{K}\\).\n\nThe training objective is \\(\\mathcal{L} = \\mathcal{L}_{\\text{inv}} + \\lambda\\,\\mathcal{L}_{\\text{SIGReg}}\\). The invariance term is the mean squared error between each local embedding and the global one; minimized in isolation it admits the constant solution. SIGReg excludes that solution by constraining the embedding distribution to an isotropic Gaussian, from which any collapsed configuration is maximally distant. By the Cramér–Wold theorem the constraint reduces to univariate goodness-of-fit tests along random directions, evaluated with the Epps–Pulley statistic at cost linear in batch size and embedding dimension.\n\nThe trade-off weight \\(\\lambda\\) constitutes the objective's only hyperparameter. The trainable architecture comprises the encoder and projector alone; neither a predictor network nor a target encoder is instantiated.\n\n## Token dropping\n\nA fraction \\(\\rho\\) of the patch tokens of each view is discarded uniformly at random after patch embedding, and the retained tokens constitute the encoder's sole observation of the clip. Were this an approximation adopted for efficiency, accuracy would be expected to decline as \\(\\rho\\) increases. The converse is observed: ImageNet-1K accuracy rises monotonically, from \\(33.9\\%\\) at \\(\\rho = 0\\) to \\(47.6\\%\\) at \\(\\rho = 0.95\\).\n\nToken dropping therefore fulfills two functions. It reduces the cost of each forward pass by a factor of up to \\((1-\\rho)^{-1}\\), and it constitutes a stochastic augmentation under which the clip-level embedding must be inferable from a sparse, randomly located sample of the clip.\n\n39 of 784 tokens\n\nThe spatial arrangement of the retained set is equally consequential. A tube variant retaining identical spatial locations in every frame attains \\(39.6\\%\\) against \\(50.7\\%\\) for uniform random dropping, reversing the ordering established in masked video modeling.\n\nTwo further observations follow. Accuracy increases with the number of local views, from \\(47.6\\%\\) at \\(V = 4\\) to \\(50.2\\%\\) at \\(V = 10\\), so the results reported below do not exhaust the method. Temporal patch aggregation at the input, conventional in video transformers, proves unnecessary: at a matched token budget, per-frame tokenization attains \\(50.7\\%\\) against \\(47.4\\%\\) on ImageNet-1K and \\(30.4\\%\\) against \\(28.8\\%\\) on Something-Something-v2.\n\n## Block-causal attention\n\nBecause the objective imposes no asymmetry between branches, the attention topology of the encoder is unconstrained. We adopt a block-causal pattern in which patch tokens attend bidirectionally within their frame and causally across frames, such that each frame representation is a function of the current and preceding frames alone. Since causal masking removes future tokens from the receptive field of every token, a reduction in representation quality might reasonably be anticipated. No such reduction is observed.\n\n**current and past frames**\n\n| Attention | IN1K top-1 |\n|---|---|\n| Bidirectional | 50.7 |\n| Block-causal | 51.2 |\n\nFrozen attentive probe; both configurations use \\(\\tau = 1\\), \\(\\rho = 0.95\\), \\(V = 4\\), and uniform random dropping.\n\n## Patch-level representations\n\nThe objective supervises the clip-level `[cls]`\n\ntoken exclusively; patch tokens receive no direct\nsupervision and no auxiliary dense loss is applied. Semantically organized patch representations nevertheless\nemerge. V-JEPA 2.1 obtains comparable structure through an explicitly introduced patch-level loss, and\nV-JEPA 2, trained without such a loss, exhibits no comparable token-level organization.\n\nThe same behavior holds under a query-based readout. Cosine similarity between a patch placed on an object and all remaining patch tokens stays confined to that object rather than diffusing across the frame, indicating representations that are spatially precise as well as semantically grouped.\n\nBecause the encoder is block-causal, these maps are computed from the current and preceding frames alone, so the correspondence they express is maintained as the scene moves rather than recovered by attending forward in time.\n\n## Empirical comparison\n\nTo eliminate confounds in pretraining data, schedule, and compute, all baselines are retrained on an identical \\(20\\%\\) subsample of K710 using their official implementations, for \\(240\\) epochs at an effective batch size of \\(3{,}072\\), with every encoder probed on an equal number of tokens.\n\nAcross all three encoder sizes LeVJEPA attains accuracy comparable to or exceeding V-JEPA 2 at a fraction of the total pretraining compute, the advantage ranging from \\(5.6\\times\\) at ViT-L to \\(20.8\\times\\) at ViT-S. At ViT-B the two methods are separated by less than one accuracy point, at \\(4.8\\) against \\(36.4\\) ExaFLOPs. Under a fixed total FLOP budget, the reduced per-sample cost admits a proportionally longer schedule of \\(1{,}085\\) epochs at \\(V = 10\\).\n\n| Method | IN1K | SSv2 | K400 |\n|---|---|---|---|\n| VideoMAEv2 | 53.4 | 43.6 | 37.4 |\n| V-JEPA 2 | 51.6 | 42.5 | 40.7 |\n| LeVJEPA | 61.0 | 40.4 | 44.6 |\n\nViT-B encoders at equal total pretraining FLOPs, evaluated frozen. IN1K and SSv2 report attentive-probing top-1 accuracy; K400 reports linear probing on mean-pooled tokens, a strictly weaker adaptation.\n\n### Comparison with image pretraining\n\nImage-based self-supervised learning has hitherto constituted the stronger paradigm for appearance-centric transfer. Training DINOv2 with its official implementation on individual frames of the same video data (\\(11.7\\)M frame samples over \\(11{,}400\\) optimizer steps) at equal total FLOPs, the image-pretrained encoder retains an advantage of \\(3.1\\) points on ImageNet-1K, while the video-pretrained encoder attains nearly twice its accuracy on Something-Something-v2.\n\n| Method | IN1K | SSv2 |\n|---|---|---|\n| DINOv2 | 53.8 | 16.9 |\n| LeVJEPA | 50.7 | 30.4 |\n\nViT-B encoders at equal total FLOPs on identical source data; frozen attentive probes.\n\n## Computational requirements and data scaling\n\nThe reduced per-sample cost lowers the hardware threshold for pretraining. A ViT-Tiny trained for \\(12\\) hours on a single consumer GPU on eight unannotated Walking Tours videos, approximately \\(620\\)k frames, improves from \\(8.9\\%\\) to \\(25.2\\%\\) ImageNet-1K top-1 under frozen evaluation.\n\nRelaxing the \\(20\\%\\) restriction, a ViT-L/16 pretrained for \\(100\\) epochs on the union of K710, Something-Something-v2, Walking Tours, and the PE Video Dataset attains \\(67.5\\%\\) on ImageNet-1K and \\(55.0\\%\\) on Something-Something-v2 under frozen attentive probing, within a shorter schedule and without modification to the objective or its single hyperparameter.\n\n## Cite\n\n```\n@misc{kuhn2026levjepaefficientscalable,\n      title={LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics}, \n      author={Lukas Kuhn and Lucas Maes and Giuseppe Serra and Quentin Le Lidec and Yann LeCun and Randall Balestriero and Florian Buettner},\n      year={2026},\n      eprint={2608.27395},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2608.27395}, \n}\n```\n\n", "url": "https://wpnews.pro/news/levjepa-efficient-and-scalable-video-pretraining-without-the-heuristics", "canonical_source": "https://levjepa.github.io/", "published_at": "2026-08-29 16:30:23+00:00", "updated_at": "2026-08-29 16:48:44.132599+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "computer-vision"], "entities": ["LeVJEPA", "V-JEPA 2", "German Cancer Research Center", "German Cancer Consortium", "Goethe University Frankfurt", "Mila", "Université de Montréal", "Brown University"], "alternates": {"html": "https://wpnews.pro/news/levjepa-efficient-and-scalable-video-pretraining-without-the-heuristics", "markdown": "https://wpnews.pro/news/levjepa-efficient-and-scalable-video-pretraining-without-the-heuristics.md", "text": "https://wpnews.pro/news/levjepa-efficient-and-scalable-video-pretraining-without-the-heuristics.txt", "jsonld": "https://wpnews.pro/news/levjepa-efficient-and-scalable-video-pretraining-without-the-heuristics.jsonld"}}