What exactly does word2vec learn?

BAIR · 10 months ago

Researchers developed a closed-form mathematical theory explaining how word2vec learns word embeddings, proving that the algorithm reduces to PCA on a matrix defined by corpus co-occurrence statistics. Word2vec achieves 68% accuracy on analogy completion benchmarks while learning discrete, sequential linear concepts that correspond to interpretable topics. The theory enables predicting learned features beforehand from corpus statistics and provides insights into how neural language models develop meaningful representations.