NLP for Supervised Learning - A Brief Survey

Eugene Yan spent his downtime doing something unusual for an ML practitioner: reading old papers in order, not for a project deadline, but just to understand how we got here. The result is a chronological tour of NLP's building blocks, starting with RNNs in 1985 and ending at T5 in 2019, and it's a useful reminder that today's language models didn't spring out of nowhere. They're stacked on three decades of incremental fixes.

The early story is about memory. RNNs let networks process sequences by feeding each hidden state back into itself, but they choked on long sentences because gradients either vanished or exploded during backpropagation. LSTMs, introduced in 1997, fixed this with forget, input, and output gates that decide what to keep and what to drop — though it took until 2015 for them to actually ship in products like Siri, Alexa, and Google Voice. GRUs simplified that gating in 2014, trading a bit of expressiveness for faster training.

Then came the embedding era. Word2vec, in 2013, turned words into dense vectors learned from raw text, using either CBOW or Skip-gram, with Skip-gram doing notably better on rare words because it doesn't let common words drown out the rare ones in training. GloVe followed in 2014 with a global co-occurrence matrix instead of Word2vec's sliding window, and oddly enough the two methods converged on similar representations despite starting from different math. ELMo, in 2018, pushed further by generating context-sensitive embeddings via a bidirectional LSTM, so the word