21. From Images to Video: V-JEPA

I-JEPA operates over spatial structure:

\[H\times W\]

Video adds another axis:

\[T\times H\times W\]

V-JEPA uses spatio-temporal tubelets.

A typical tubelet may cover:

\[2\times16\times16\]

corresponding to two frames in time and a 16×16 spatial region.

Masking becomes spatio-temporal as well.

The fundamental objective remains:

Predict the latent representation of masked video regions from visible context.

22. V-JEPA 2: Scaling Video Self-Supervision

V-JEPA 2 largely preserves the V-JEPA prediction objective while increasing the scale of training.

The model is scaled along several axes:

  • larger datasets,
  • larger encoders,
  • longer training,
  • longer video clips,
  • higher spatial resolution.

The paper scales the encoder to approximately one billion parameters and trains on over one million hours of video.

23. Progressive Video Training

Long, high-resolution video is expensive.

A 64-frame 384×384 setup throughout the entire training process would create a massive compute burden.

V-JEPA 2 therefore uses progressive training.

Most training is performed with shorter and lower-resolution clips. During the final cooldown stage, spatial resolution and video duration are increased.

Conceptually:

\[\text{short + low resolution} \rightarrow \text{longer + high resolution}\]

The paper reports approximately an 8× reduction in GPU time for the 64-frame, 384-resolution regime compared with full-resolution training throughout.

24. What Does Time Add to a Representation?

An individual image answers questions such as:

What is present?

Video can additionally answer:

What is changing?

How is it moving?

What might happen next?

Motion understanding

Can the representation distinguish how objects move over time?

Example:

Is the hand moving upward or downward?

Temporal context

Does the representation encode the ordering and relationships between events?

For example:

\[A\rightarrow B\rightarrow C\]

is generally not equivalent to:

\[C\rightarrow B\rightarrow A\]

Action anticipation

Given observed video context, does the representation contain enough information to predict what action is likely to happen next?

These evaluations help test whether the representation contains genuine temporal information rather than only strong static appearance features.

25. Action Anticipation Is Not V-JEPA 2-AC

In ordinary V-JEPA 2 evaluation:

\[\text{past video} \rightarrow \text{future action label}\]

The action is the prediction target.

In V-JEPA 2-AC:

\[(z_t,a_t) \rightarrow \hat z_{t+1}\]

The action is an input condition.

A compact way to remember the difference is:

Action anticipation: What will be done next?

Action-conditioned modeling: What will happen if I do this?

26. From Prediction to Planning: V-JEPA 2-AC

V-JEPA 2-AC turns the learned video representation into an action-conditioned latent world model.

The V-JEPA 2 encoder is frozen.

A new predictor is trained using a comparatively small amount of robot interaction data.

It learns dynamics of the form:

\[(\text{current latent state},\text{action}) \rightarrow \text{future latent state}\]

This allows the model to reason about the effects of possible robot actions without generating future video pixels.

A planning loop can evaluate candidate action sequences in latent space and choose actions whose predicted future representations move the system toward the desired goal representation.

This is model-based planning, not simply a learned feed-forward policy.

27. What Does “Zero-Shot” Mean Here?

V-JEPA 2-AC is not zero-shot in the sense that it has never seen robot interaction data.

It is post-trained using under 62 hours of DROID robot interaction video.

The relevant zero-shot claim is that the model is deployed in new laboratory environments without collecting training data from those deployment robots and without task-specific reward training.

So the safer interpretation is:

Zero-shot transfer to the deployment environment.

28. A Useful Mental Model for the Entire Progression

SimCLR / MoCo

Prevent collapse using negative examples.

\[\text{positive attraction} + \text{negative repulsion}\]

BYOL / DINO

Remove explicit negatives and use asymmetric teacher-student learning dynamics.

MAE

Leave view matching behind and predict masked pixels.

VICReg

Return to joint embedding, but make collapse prevention explicit.

\[\text{invariance} + \text{variance} + \text{covariance}\]

VISReg

Go beyond second-order decorrelation.

\[\text{scale} + \text{distribution shape} + \text{center}\]

I-JEPA

Move masked prediction into representation space.

\[\text{context} \rightarrow \text{latent target}\]

V-JEPA

Extend latent prediction into space and time.

\[H\times W \rightarrow T\times H\times W\]

V-JEPA 2

Scale video self-supervision.

V-JEPA 2-AC

Condition latent dynamics on actions and use them for planning.

29. The Broader Direction

These methods are not simply replacements for one another. They explore different answers to several fundamental questions.

What defines similarity?

Contrastive learning uses positive and negative examples.

Joint-embedding methods use augmented views.

Predictive methods use contextual relationships.

How should collapse be prevented?

Negative examples provide explicit repulsion.

Teacher-student methods rely on asymmetric learning dynamics.

VICReg and VISReg directly constrain representation statistics.

What should the model predict?

MAE predicts pixels.

I-JEPA predicts image representations.

V-JEPA predicts spatio-temporal representations.

V-JEPA 2-AC predicts action-conditioned future representations.

There is a visible shift from reconstructing sensory details toward modeling increasingly abstract and predictable aspects of the world.

A model that can represent a state, predict how that state changes, and reason about the effect of an action begins to look much closer to a usable world model.

Final Takeaways

1. Collapse prevention is a design problem, not a solved detail.

Different SSL families handle it through negatives, architectural asymmetry, explicit statistics, or the structure of the prediction problem.

2. Augmentation is itself an inductive bias.

SimCLR, BYOL, and DINO learn invariances defined partly by transformations chosen by the researcher.

I-JEPA explores an alternative based on contextual prediction.

3. Prediction space matters.

Predicting pixels and predicting learned representations can result in very different representations.

4. Representation geometry matters.

VICReg shows that variance and redundancy can be directly controlled.

VISReg pushes this idea further from second-order statistics toward distribution-level geometry.

5. Video introduces more than additional images.

It introduces temporal dynamics, opening the door to motion understanding, anticipation, and eventually action-conditioned world modeling.

6. Prediction and planning are becoming increasingly connected.

V-JEPA 2-AC illustrates a particularly interesting direction: learn visual representations from massive passive video data, then use a relatively small amount of interaction data to learn how actions transform those representations.

The long-term question is no longer only:

Can the model recognize what it sees?

It increasingly becomes:

Can the model represent what it sees, predict what will happen, and reason about what would happen if it acted?

References

  • Chen et al. A Simple Framework for Contrastive Learning of Visual Representations (SimCLR), ICML 2020.
  • He et al. Momentum Contrast for Unsupervised Visual Representation Learning (MoCo), CVPR 2020.
  • Grill et al. Bootstrap Your Own Latent (BYOL), NeurIPS 2020.
  • Caron et al. Emerging Properties in Self-Supervised Vision Transformers (DINO), ICCV 2021.
  • He et al. Masked Autoencoders Are Scalable Vision Learners, CVPR 2022.
  • Bardes et al. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, ICLR 2022.
  • Assran et al. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, CVPR 2023.
  • Assran et al. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning, 2025.
  • Wu et al. VISReg: Variance-Invariance-Sketching Regularization for JEPA Training, 2026.