OlmPool: How small architectural choices compound to undermine long context extension

Allen Institute (AI2) · 4 months ago · 11

Researchers found that four architectural choices in language models—QK normalization, grouped-query attention, sliding window attention, and shorter pretraining context length—individually have modest negative effects on long context performance but combine to reduce benchmark scores by up to 47%. The study used OlmPool, a suite of 26 7-billion-parameter models trained on 140 billion tokens with identical data but different architectures, to isolate these effects. The findings show that Llama 3's long context capabilities come primarily from architecture rather than training data, meaning context extension recipes validated on Llama may not transfer to other model families without adjustment.