4. DINO: Self-Distillation With a Momentum Teacher
DINO keeps the teacher-student direction but changes the learning target.
Instead of directly regressing one embedding vector toward another, the student is trained to match the teacher’s output distribution.
The teacher is updated using an EMA of the student.
Two mechanisms are especially important.
Centering
Teacher logits are centered using a moving average. This prevents a small number of dimensions from dominating all examples.
Sharpening
A low teacher temperature produces sharper target distributions, countering the tendency toward uniform outputs.
Multi-crop training
DINO also uses multi-crop augmentation.
Instead of producing only two views, it can generate:
- two large global crops
- several smaller local crops
The teacher processes global views, while the student processes both global and local views.
This encourages local-to-global consistency:
A small local crop should still produce an output compatible with the semantic structure learned from the larger global view.