Contrastive Representation Learning
Lilian Weng
Contrastive representation learning aims to learn embedding spaces where similar samples cluster together and dissimilar ones remain distant, applicable to both supervised and unsupervised settings. Key loss functions include contrastive loss (Chopra et al. 2005), triplet loss (Schroff et al. 2015), lifted structured loss (Song et al. 2015), N-pair loss (Sohn 2016), noise contrastive estimation (Gutmann & Hyvarinen 2010), InfoNCE (van den Oord et al. 2018), and soft-nearest neighbors loss (Salakhutdinov & Hinton 2007). Modern contrastive objectives process multiple positive and negative sample pairs within a batch to improve computational efficiency and model performance.
Why it matters
The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.
