The Transformer Family

Lilian Weng · 6 years ago

An article explains the Transformer architecture and describes improvements to the vanilla Transformer model for longer attention spans, reduced memory consumption, and better performance on various tasks. The vanilla Transformer uses multi-head self-attention with fixed segment lengths of L tokens and 6 stacked encoder-decoder layers. Transformer-XL and other enhanced variants address limitations by reusing hidden states between segments and adopting new positional encoding methods.