The Big LLM Architecture Comparison

Seven years after GPT-2, you'd expect LLM architecture to look unrecognizable. It doesn't, really. A new deep-dive comparison from the Ahead of AI newsletter walks through the flagship open models of 2025 — DeepSeek V3/R1, OLMo 2, and a growing list of others — and the honest takeaway is that everyone is still building on the same transformer bones. What's changed is the plumbing: how attention handles memory, how feedforward layers scale, and where you slap the normalization.

DeepSeek gets the most attention, and for good reason. Its V3/R1 lineage combines two tricks that made a 671-billion-parameter model surprisingly cheap to run. Multi-Head Latent Attention compresses keys and values into a smaller space before caching them, undoing the compression only at inference time — a bit more math per step, but a lot less memory pressure. DeepSeek's own ablation studies suggest this actually beats standard multi-head attention on quality, not just efficiency, which is why the team picked it over the now-common Grouped-Query Attention. Then there's the Mixture-of-Experts setup: 256 experts sitting inside each MoE block, but only nine ever fire for a given token — one shared expert plus eight chosen by a router. So instead of touching all 671 billion parameters, DeepSeek only activates about 37 billion per step. That shared expert isn't a DeepSeek invention (it traces back to 2022's DeepSpeedMoE work), but it's a smart division of labor: common patterns get learned once, freeing the other experts to specialize.

OLMo 2, from the nonprofit Allen Institute for AI, is a different kind of interesting. Nobody's putting it at the top of a leaderboard, but it's the most transparent model family around — full training data, full code, detailed writeups — and back in January, before Llama 4 and Gemma 3 showed up, it was sitting right on the compute-to-performance frontier. Architecturally it's almost boringly close to the original GPT design, still using plain Multi-Head Attention instead of GQA or MLA. The real tweak is where they put the RMSNorm layers. Instead of the now-standard Pre-Norm placement (before attention and feedforward blocks), OLMo 2 reverts to a Post-Norm style — normalization after the sublayers, but still inside the residual stream — paired with QK-Norm, an extra RMSNorm applied to queries and keys before RoPE is applied. The combination visibly smooths out training stability, though the paper doesn't cleanly separate how much credit belongs to the reordering versus the QK-Norm addition.

What comes through across the piece, even before it gets to Gemma or Llama 4, is that the industry has converged on a shared toolkit — RoPE instead of absolute positions, SwiGLU instead of GELU, sparse MoE layers, and increasingly baroque normalization placement — and everyone is just remixing the same handful of levers. Nobody's throwing out the transformer. They're just tuning where the memory savings happen and where the stability comes from, one normalization layer and one expert router at a time.