The Transformer Family

Lilian Weng's post is one of those reference pieces that quietly became load-bearing infrastructure for anyone trying to understand how Transformers evolved past the 2017 "Attention Is All You Need" blueprint. It starts where her earlier attention post left off and asks a practical question: how do you take a model that's great at short sequences and make it handle longer context, use less memory, and still train efficiently?

The piece opens with the mechanics everyone glosses over — scaled dot-product attention, the multi-head split-and-concatenate trick, and the encoder-decoder stack with its six identical layers each. It's a good refresher on why positional encoding exists at all: self-attention has no built-in sense of order, so Vaswani's team bolted on sinusoidal encodings, alternating sine and cosine waves across dimensions, to give the model a notion of sequence position. Weng also flags the lesser-known alternative, learned positional embeddings from Gehring's 2017 work, which just lets the model memorize position vectors instead of computing them.

Where the post gets genuinely useful is in the fixes people built on top of vanilla Transformers. Al-Rfou's 2018 character-level model added auxiliary losses so intermediate layers and intermediate positions all had to make correct predictions too, not just the final output — a trick that reportedly beat LSTMs at the time. Then there's Adaptive Computation Time, borrowed from Graves' 2016 recurrent-network work, which lets a model decide dynamically how many computation steps a given input deserves, with a