9. VISReg: Decoupling Scale and Shape

VISReg replaces VICReg’s covariance regularization with a stronger distribution-level objective while preserving explicit scale control.

Its regularization is decomposed into:

\[L_{\text{Reg}} = \lambda_{\text{scale}}L_{\text{scale}} + \lambda_{\text{shape}}L_{\text{shape}} + \lambda_{\text{center}}L_{\text{center}}\]

The key conceptual separation is:

Scale asks how much the representation spreads.
Shape asks how that spread is geometrically organized.

Scale Regularization

VISReg encourages each feature dimension to have standard deviation close to one:

\[L_{\text{scale}} = \frac{1}{D} \sum_j (1-\sigma_j)^2\]

This keeps the representation away from collapse.

The number one is not semantically special. It simply provides a well-defined reference scale.

10. Shape Regularization With Sliced Wasserstein Distance

VISReg wants the full geometry of the embedding distribution to resemble a well-behaved isotropic Gaussian.

Direct high-dimensional optimal transport would be expensive, so VISReg uses Sliced Wasserstein Distance.

Take a high-dimensional embedding:

\[z\in\mathbb{R}^{D}\]

Choose a random direction:

\[w_k\]

and project:

\[p_k=z^\top w_k\]

For an entire batch, this creates a one-dimensional distribution.

The algorithm then:

  1. projects embeddings onto a random direction,
  2. sorts the projected values,
  3. obtains empirical quantiles,
  4. compares them against standard Gaussian quantiles,
  5. repeats this for many random directions.

Conceptually:

If the embedding distribution looks Gaussian from many different one-dimensional perspectives, its global geometry is being pushed toward the isotropic Gaussian prior.

Why an isotropic Gaussian?

This does not mean semantic categories are assumed to be Gaussian.

The Gaussian is used as a geometric prior because it is centered, balanced, directionally symmetric, and mathematically convenient.

The semantic information still comes from the self-supervised invariance or prediction objective.

11. Why VISReg Uses Stop-Gradient

VISReg normalizes its centered embeddings using their standard deviation:

\[\tilde Z = \frac{\hat Z} {\operatorname{sg}(\sigma)+\epsilon}\]

The stop-gradient prevents the shape objective from modifying the scale through the denominator.

This decouples scale optimization from shape optimization:

  • scale loss controls magnitude,
  • shape loss controls geometry.

The stop-gradient is not being used as the primary anti-collapse mechanism as in BYOL-style training.

12. What Does VISReg Improve?

VISReg should not be interpreted as:

“VICReg is wrong and VISReg always performs better.”

Its stronger claims are about:

  • distribution-level control,
  • scale/shape decomposition,
  • strong collapse-region gradients,
  • favorable scaling characteristics,
  • robustness on low-quality and long-tailed datasets,
  • strong out-of-distribution generalization.

It does not necessarily dominate every method on in-domain linear probing.

13. A Different Direction: From Regularization to Prediction

VICReg and VISReg ask:

How should the embedding space be organized?

JEPA asks:

What should the representation be trained to predict?

MAE showed that masking is a strong self-supervised signal, but its target is still pixel space.

JEPA asks whether the prediction target itself can be more abstract.