RePo: Language Models with Context Re-Positioning

Sakana AI · 37 ● 2 sources

Sakana AI introduced RePo, a method that allows language models to dynamically reorganize their input context based on content relevance rather than processing information in a fixed linear sequence. The approach outperforms standard encodings on noisy contexts, structured data, and long-range dependencies while maintaining competitive general performance. This enables models to actively reshape their attention patterns to match problem structure rather than treating physical proximity as semantic relevance.