What exactly does word2vec learn?

For a technique that's been around since 2013 and inspired half the language modeling field, word2vec has kept its inner workings surprisingly opaque. Everyone knew it produced those famous linear analogies — king minus man plus woman equals queen — but nobody had a clean mathematical account of why that happens or what the model is actually doing during training. A new paper from Dhruva Karkada and collaborators, posted on the BAIR blog, closes that gap with a closed-form theory that's almost embarrassingly tidy.

The core finding: under a few mild, data-independent conditions — small initial weights, tiny gradient steps, a quartic approximation near the origin, and a specific relationship between hyperparameters — word2vec's training dynamics collapse into unweighted least-squares matrix factorization. Solve that, and the learned embeddings turn out to be nothing more exotic than the top eigenvectors of a particular matrix built from corpus statistics: co-occurrence probabilities and unigram frequencies, run through a simple normalized ratio. Diagonalize that matrix on Wikipedia text and the eigenvectors sort themselves into recognizable topics — celebrity biographies first, then government and municipal administration, then geography — in descending order of importance.

What's genuinely striking is the learning pattern itself. Rather than smoothly improving, word2vec trained from near-zero initialization learns in discrete jumps, each one adding a new orthogonal dimension to the embedding space and producing a visible step down in the loss curve. Karkada compares it to the disorienting first weeks of studying an unfamiliar branch of math, where jargon blurs together until, gradually, distinct concepts peel apart and click into place. Once a subspace is learned here, it doesn't rotate or get overwritten — it just sits there as a stable feature while the next concept gets carved out.

The approximations hold up better than you'd expect. On the standard word-analogy benchmark, real word2vec scores 68% accuracy; the simplified theoretical model scores 66%; the older PPMI method, a classical alternative, manages only 51%. None of the four approximations touch the actual data distribution, which the authors flag as the real strength here — the theory predicts which features get learned purely from corpus statistics and training hyperparameters, without assuming anything about the shape of the underlying language data.

The team also used the framework to poke at how abstract binary concepts — masculine versus feminine, past versus future — emerge during training, modeling the process with a spiked random matrix approach. Early training rounds are dominated by genuine semantic signal, but push training too far and noise starts creeping in, degrading the very linear representations that made word2vec's analogy trick famous in the first place.