1. Contrastive Learning: Prevent Collapse With Negatives

SimCLR

SimCLR begins with a very intuitive objective.

Take an image (x), create two independent augmented views:

\[x_i = t_i(x), \qquad x_j = t_j(x)\]

and encode both:

\[h_i=f_\theta(x_i), \qquad h_j=f_\theta(x_j)\]

A projection head then maps the encoder representations into a contrastive embedding space:

\[z_i=g(h_i), \qquad z_j=g(h_j)\]

The objective encourages the two views of the same image to have similar embeddings while embeddings from different images are pushed apart.

The NT-Xent / InfoNCE-style objective can be written roughly as:

\[\ell_{i,j} = -\log \frac {\exp(\operatorname{sim}(z_i,z_j)/\tau)} {\sum_k\exp(\operatorname{sim}(z_i,z_k)/\tau)}\]

where (\tau) is a temperature parameter.

If every image were mapped to the same vector, the model would no longer be able to distinguish the positive pair from the many negative examples, so collapse is naturally discouraged.

The cost of contrastive learning

The strength of the task depends heavily on the number of negative examples. More negatives mean the model must identify its positive partner among more alternatives. This makes large batches useful, but also expensive.

The practical question becomes:

Do we really need to place thousands of negatives inside the current mini-batch?