2. MoCo: Decoupling the Number of Negatives From Batch Size
Momentum Contrast keeps the same basic contrastive objective but changes how negative examples are maintained.
Instead of relying only on the current mini-batch, MoCo stores embeddings from previous batches inside a queue.
A query representation (q) is compared against one positive key (k^+) and a large dictionary of negative keys.
The major innovation is therefore not the loss. It is the negative dictionary.
Why the momentum encoder?
Embeddings inside the queue were produced at different points in training. If the encoder changed too quickly, old and new embeddings would belong to increasingly different representation spaces.
MoCo therefore updates its key encoder slowly:
This creates a slowly evolving target network and keeps the dictionary relatively consistent over time.
SimCLR and MoCo largely solve the same contrastive problem. MoCo changes how the negatives are stored and updated.