6. VICReg: Make Collapse Prevention Explicit

BYOL and DINO demonstrated that non-contrastive joint-embedding learning is possible, but collapse prevention still relied on mechanisms such as predictor asymmetry, EMA teachers, stop-gradient, centering, and sharpening.

VICReg asks:

Can we explicitly define what a healthy embedding space should look like?

The loss combines three objectives:

\[L = \lambda L_{\text{inv}} + \mu L_{\text{var}} + \nu L_{\text{cov}}\]

Invariance

Two augmented views of the same image should remain close:

\[L_{\text{inv}} = \|z-z'\|^2\]

But invariance alone has a trivial solution:

\[z_1=z_2=\cdots=z_N=c\]

If every example maps to the same representation, positive pairs are perfectly aligned.

Variance

For every embedding dimension, VICReg encourages the batch standard deviation to remain above a threshold:

\[\sigma_j \geq \gamma\]

This directly prevents complete collapse.

If every sample maps to the same value in one dimension:

\[\sigma_j \rightarrow 0\]

and the variance regularization becomes active.

Do not merely hope that the architecture avoids collapse. Explicitly require the embedding dimensions to remain active.

Why standard deviation instead of variance?

Suppose the loss directly constrained:

\[\operatorname{Var}(x)\]

The gradient of variance with respect to a sample roughly contains:

\[x_i-\bar x\]

As the representation approaches collapse:

\[x_i\rightarrow \bar x\]

this gradient approaches zero.

VICReg instead uses:

\[\sigma = \sqrt{\operatorname{Var}(x)+\epsilon}\]

whose derivative introduces a factor proportional to:

\[\frac{1}{\sqrt{\operatorname{Var}(x)+\epsilon}}\]

This makes the regularization more useful in the low-variance regime, where a strong corrective signal is needed.

7. Covariance: Reducing Feature Redundancy

Variance prevents dimensions from becoming constant, but active feature dimensions may still carry redundant information.

For example:

\[z_2\approx2z_1\]

Both dimensions vary, but they encode nearly the same signal.

VICReg computes the covariance matrix of the embeddings and penalizes the off-diagonal terms:

\[C_{ij}\rightarrow0 \qquad i\neq j\]

A useful interpretation is:

Variance says: every dimension should carry some variation.
Covariance says: those dimensions should not all carry the same variation.

This reduces feature redundancy.

Decorrelated does not mean independent

If

\[\operatorname{Cov}(X,Y)=0\]

there is no detected linear second-order dependency between (X) and (Y), but this does not imply statistical independence.

For example, if:

\[Y=X^2\]

then (Y) may be completely determined by (X), yet under a symmetric distribution their covariance can still be zero.

This limitation directly motivates VISReg.

8. From VICReg to VISReg: Is Decorrelation Enough?

Covariance controls second-order statistics, but two distributions can have:

\[\mu_A=\mu_B\]

and

\[\Sigma_A=\Sigma_B\]

while still having radically different shapes:

\[p_A(z)\neq p_B(z)\]

One could be Gaussian, another could form a ring, and another could form an X-shaped structure.

This exposes the key limitation:

Decorrelation does not uniquely determine the geometry of the embedding distribution.

VISReg begins from this observation.