Which tokens does a hybrid model predict better?

Benchmarks are great for bragging rights but terrible for explaining why one model architecture actually wins. The Allen Institute for AI ran into that problem with Olmo Hybrid, its recurrence-plus-attention model, and decided to dig into the guts of the thing rather than just report a leaderboard score.

They took two 7-billion-parameter models built as identically as possible — same data, same tokenizer, same training recipe — except one was a standard transformer (Olmo 3) and the other swapped most attention layers for recurrent ones (Olmo Hybrid). Then they fed both models articles, Wikipedia pages, books, scientific papers, even raw Python and HTML, and tracked which model predicted each next token more accurately. Because everything else was held constant, any gap in performance came down to architecture alone.

The pattern that emerged was sharper than a single aggregate score would ever show. Olmo Hybrid crushed the transformer on content words — nouns, verbs, adjectives, the words that actually carry meaning — with a loss gap around 0.04. On glue words like "the" or "of," the advantage shrank to about half that, which makes sense since a model can often guess those from grammar alone. But the real surprise was where the hybrid's edge vanished entirely: closing brackets, and any token that was simply a verbatim repeat of something earlier in the text. Attention, it turns out, is basically built for exact lookup — reach back, grab the token, done. Recurrent layers compress everything into a fixed-size memory, so they're great at tracking things that evolve over a passage but bad at retrieving an exact word from three paragraphs ago.

AI2 pushed this further with three smaller 1B models — a transformer, a hybrid, and a pure recurrent model with zero attention. The pure recurrent version, lacking any lookup mechanism, fell noticeably behind on repeated text, while both the hybrid and the transformer handled copying fine. On fresh, meaning-heavy content, though, the hybrid pulled ahead of everyone, recurrence again giving it an apparent edge at tracking state as it reads.

The upshot AI2 draws is that overall loss numbers hide more than they reveal. If you want to know what an architecture is actually good at, you have to slice the evaluation by token type — content words versus function words, fresh text versus repeats — because averaging across everything smears out exactly the differences that matter for deciding what a hybrid model should be built to do next.