The Transformer Family Version 2.0
Lil'Log
Lilian Weng dropped a massive update to her classic 'Transformer Family' blog post, doubling its length. It's a refreshed map of every major architecture tweak researchers have made since 2020.
Based on reporting by Lil'Log — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Three years is a long time in AI research, and Lilian Weng's original rundown of Transformer variants was starting to show its age. So she went back in and basically rebuilt the thing from scratch, keeping the old content as a foundation but restructuring the sections and folding in a wave of newer papers. The result, which she's calling Version 2.0, is roughly twice as long as the 2020 original and reads less like an update and more like a full second edition.
The piece starts where any serious Transformer explainer has to: notation. Weng lays out the full alphabet soup of symbols that show up across attention papers — model dimension d, head count h, sequence length L, layer count N, and the various weight matrices for queries, keys, and values. It's dry stuff, but anyone who's tried to compare two attention papers with slightly different variable naming knows why this matters. Getting the notation locked down early makes the rest of the comparison actually legible.
From there she moves into the basics, tracing the vanilla Transformer from Vaswani et al.'s 2017 encoder-decoder design through the split that came later, where researchers found you didn't need both halves. BERT proved an encoder-only setup could crush language understanding tasks, while GPT went the other direction, showing a decoder-only stack was enough to drive generative language modeling. That fork, encoder-only versus decoder-only versus the original full encoder-decoder, still defines how most people categorize models today.
What makes this update worth flagging isn't the history lesson, though — it's the sheer accumulation of architectural tweaks Weng is now cataloging that didn't exist, or weren't mainstream, back in 2020. Three years of efficiency hacks, attention approximations, and positional encoding schemes have piled up, and having one document that organizes all of it by category rather than by publication date is genuinely useful for anyone trying to track how the field actually moved, as opposed to which paper got the most Twitter attention that month.
My take — AI-written commentary, not fact-checked reporting
I'll say it: reference posts like this age better than most 'breakthrough' papers, because they force someone to sit down and actually reconcile competing claims instead of just hyping the newest one. Weng doubling the length in three years is itself a data point about how frantic Transformer research has gotten, and it's exactly the kind of unglamorous synthesis work the field doesn't reward enough.
Read more about this at: Lil'Log