Which tokens does a hybrid model predict better?

Allen Institute (AI2) · 3 weeks ago

Researchers compared Olmo 3 (a transformer) and Olmo Hybrid (a hybrid architecture) by analyzing how well each predicted different token types, finding that hybrid models excel at predicting content words like nouns and verbs with a loss gap of 0.04 compared to 0.02 on function words, while transformers maintain their advantage on tokens that repeat verbatim from earlier in the passage. The study used regression analysis across passages of prose, code, and structured text to isolate architecture-specific strengths. This fine-grained token-level analysis suggests that hybrid architectures warrant further development and that single overall loss metrics are insufficient for comparing different model architectures.