5. MAE: Stop Matching Views and Reconstruct Missing Content

Masked Autoencoders take a different route.

Instead of creating two augmented views and forcing their representations to agree, MAE masks a large fraction of the image.

Typically around (75%) of the patches are removed.

Only visible patches enter the encoder. A lightweight decoder receives the encoded visible patches plus mask tokens for missing positions and reconstructs the missing pixels.

The objective is roughly:

\[L = \frac{1}{|M|} \sum_{i\in M} \|\hat x_i-x_i\|^2\]

This changes the collapse problem. A constant representation cannot reconstruct the different pixel targets of many different images.

MAE therefore changes the learning problem itself:

Instead of defining invariance between different views, predict the content that has been removed.

But this introduces another question:

Do we really need to reconstruct every pixel?