14. I-JEPA: Predict Representations, Not Pixels

I-JEPA stands for Image-based Joint-Embedding Predictive Architecture.

The central idea is:

Given a visible context region, predict the learned representation of hidden target regions.

Instead of:

\[\text{visible context} \rightarrow \text{missing pixels}\]

as in MAE, I-JEPA performs:

\[\text{visible context} \rightarrow \text{missing target representation}\]

I-JEPA changes the prediction space.

15. Why Predict in Latent Space?

Pixel prediction forces the model to capture everything required for accurate reconstruction:

  • exact texture,
  • illumination,
  • fine color differences,
  • high-frequency background details.

Many of these may be irrelevant for downstream semantic understanding.

Suppose the hidden region contains a dog’s head.

Pixel prediction asks:

What exact RGB values belong here?

Latent prediction asks:

What representation would the target encoder produce for this region in the context of the full image?

The I-JEPA paper directly tests this hypothesis.

When target-encoder representations are used, low-shot linear evaluation reaches 66.9 Top-1, while using pixel targets drops the result to 40.7.

This provides strong empirical evidence that the target space itself matters.

16. I-JEPA Architecture

I-JEPA contains three major components.

Context encoder

The context encoder sees only visible image regions:

\[x_{\text{context}} \rightarrow f_\theta \rightarrow s_x\]

Target encoder

The target encoder processes the full image:

\[y \rightarrow f_{\bar\theta} \rightarrow s_y\]

It receives no direct gradient. Its weights are updated through an exponential moving average of the context encoder.

Predictor

The predictor receives context representations plus positional mask tokens indicating which regions should be predicted.

It produces:

\[\hat s_y\]

and the loss compares this against the target encoder representation:

\[L = \|\hat s_y-s_y\|^2\]

17. Why Does the Target Encoder See the Full Image?

The mask is not applied to the target encoder input.

Instead, the complete image is encoded and the target representations are selected afterward from the encoder output.

This means a target patch representation is contextualized by the surrounding image.

A patch containing brown fur is not merely represented as:

brown local texture

but may instead be represented in the context of:

the head of the dog visible in this scene.

This encourages a higher semantic level in the prediction target.

18. Multi-Block Masking

I-JEPA also shows that mask geometry matters.

The default strategy samples multiple relatively large target blocks.

Typically:

  • four target blocks,
  • each around 15–20% scale,
  • one large context block sampled from around 85–100% scale,
  • regions overlapping the targets are removed from the context.

Target blocks can overlap one another, so their percentages should not simply be added to derive a total masking ratio.

The goal is to create a prediction problem that is neither trivial nor impossible.

Tiny randomly scattered targets may be recovered from local texture. Very large targets may leave too little useful context.

The reported masking ablation is striking: multi-block masking reaches 54.2 Top-1, while rasterized, block, and random masking are much lower.

19. I-JEPA vs BYOL

Both contain a notion of predicting representations, but their learning problems are different.

BYOL

\[\text{augmented view}_1 \rightarrow \text{representation of augmented view}_2\]

The core objective is view consistency.

I-JEPA

\[\text{visible context} \rightarrow \text{representation of hidden target region}\]

The core objective is contextual prediction.

The meaningful distinction is not whether the model has a predictor. It is:

What relationship is the model learning to predict?

20. I-JEPA vs MAE

The comparison can be summarized in one line:

\[\boxed{\text{MAE: context}\rightarrow\text{pixels}}\]
\[\boxed{\text{I-JEPA: context}\rightarrow\text{latent representation}}\]

I-JEPA introduces additional target-encoder computation but can reach strong semantic representation quality with substantially fewer pretraining epochs in the comparisons reported by the paper.