NLP for Supervised Learning - A Brief Survey

Eugene Yan · 5 years ago

This article is a chronological survey of natural language processing architectures and techniques, covering sequential models like RNNs and LSTMs (1985-2014), word embedding methods including Word2vec and GloVe (2013-2016), context-aware embeddings like ELMo (2018), the Transformer attention mechanism (2017), and pre-training approaches such as ULMFiT and GPT (2017-2019). Key milestones include LSTM's introduction in 1997, Word2vec's unsupervised learning breakthrough in 2013, and the Transformer's parallel processing capability in 2017, which enabled more efficient training of large language models. These developments fundamentally changed how NLP systems represent and process text, moving from one-hot encoding to learned embeddings to contextual representations across increasingly sophisticated architectures.