Generalized Language Models

Lilian Weng · 7 years ago

A comprehensive overview of generalized pre-trained language models like BERT, GPT, and others that achieve strong performance across diverse NLP tasks without requiring labeled pre-training data. The article tracks the evolution of these models from 2018 through 2021, including releases of GPT-2, GPT-3, RoBERTa, T5, and other variants. This approach enables scaling language model training to substantially larger datasets compared to supervised learning methods used in computer vision.