Language Modeling Reading List (to Start Your Paper Club)

Eugene Yan · 2 years ago · 42

A curated reading list of fundamental papers in language modeling was compiled to support a weekly paper club, covering 50+ papers from attention mechanisms to speech recognition with one-sentence summaries for each. The list spans from foundational work like "Attention Is All You Need" (2017) through recent papers like "Llama Guard" and "Self-Rewarding Language Models," organized thematically across pretraining, scaling, retrieval, fine-tuning, and safety. The collection enables groups to systematically study language model development by pooling shared knowledge and discussion rather than learning individually.