The Transformer Family Version 2.0

Lilian Weng · 3 years ago

A comprehensive technical post updated Transformer architecture documentation with improvements proposed over the past three years, expanding the original 2020 reference material to approximately twice its length. The updated version 2.0 includes restructured sections, mathematical notations for key components like attention matrices and weight parameters, and coverage of encoder-decoder variants including BERT and GPT models. The expanded resource now serves as a more complete reference for understanding modern Transformer architecture variations and their mathematical foundations.