3. BYOL: Learning Without Negatives

BYOL challenges the assumption that negative samples are necessary.

It uses two branches:

  • an online network
  • a target network

Two augmented views of the same image are created. The online network processes one view and predicts the representation produced by the target network for the other view.

Conceptually:

\[\text{online}(x_1) \rightarrow \text{target representation of }x_2\]

The target network does not receive gradients directly. Its parameters are updated through an exponential moving average of the online network:

\[\theta_{\text{target}} \leftarrow \tau\theta_{\text{target}} + (1-\tau)\theta_{\text{online}}\]

BYOL demonstrated that strong representations can be learned without explicit negatives.

The interesting question is then:

Why does the model not collapse?

The answer is tied to asymmetric training dynamics created by the predictor, stop-gradient behavior, and the slowly moving EMA target.