RePo: Language Models with Context Re-Positioning

Every language model you've used so far treats text the same way: as a straight line of tokens, numbered in order, with position equaling meaning. That's fine when the important stuff sits right next to the question you're asking. It falls apart when the useful fact is buried on page 40 of a 60-page document and surrounded by filler. Sakana AI's new project, called RePo, argues that this rigid indexing is the real bottleneck, not model size or training data.

The idea borrows from Cognitive Load Theory, the psychology framework that explains why humans struggle to think clearly when key facts are scattered across noise. Sakana's team applied the same logic to transformers: if a model has to spend its limited attention capacity untangling disorganized input, it has less left over for actual reasoning. So instead of forcing the model to accept whatever order the text arrived in, RePo adds a module that learns to reassign token positions based on how relevant each piece of content actually is.

Practically, that means RePo can drag a distant but important sentence closer to where the model is focusing, and shove irrelevant filler further away, all without touching the underlying words. It's reshaping the geometry of attention on the fly, tailoring it to the specific problem rather than the raw layout of the input. Think of it as a model that can shuffle its own notes before answering, rather than reading them start to finish no matter what.

Sakana reports that this reshuffling pays off specifically in the messy scenarios that trip up standard models: noisy contexts, long-range dependencies, and structured data where relevance doesn't follow line order. And importantly, RePo isn't sacrificing baseline performance to get there, it stays competitive on general tasks while cleaning up in the harder cases. The framing Sakana uses, models that curate their own working memory instead of passively absorbing whatever order they're fed, is a small but real shift from treating context length as a brute-force scaling problem to treating it as an organization problem.