Self-Supervised Representation Learning
Lilian Weng
Self-supervised learning leverages unlabeled data by creating pretext tasks that generate labels automatically, allowing models to learn useful representations without expensive manual annotation. Various pretext tasks have been developed for images including rotation prediction, patch relationship prediction, jigsaw puzzles, feature counting, and colorization, with learned representations evaluated by fine-tuning on downstream tasks like ImageNet classification. This approach makes it possible to use the vastly larger pool of unlabeled data available on the internet to train models that learn semantic features beneficial for practical applications.
Why it matters
[Updated on 2020-01-09: add a new section on Contrastive Predictive Coding]. [Updated on 2020-04-13: add a “Momentum Contrast” section on MoCo, SimCLR and CURL.] [Updated on 2020-07-08: add a “Bisimulation” section on DeepMDP and DBC.] [Updated on 2020-09-12: add MoCo V2 and BYOL in the “Momentum Contrast” section.] [Updated on 2021-05-31: remove section on “Momentum Contrast” and add a pointer to a full post on “Contrastive Representation Learning”]
