Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
Sakana AI ● Covered by 2 sources
Sakana AI built DroPE, a trick that stretches an LLM's context window by yanking out its positional embeddings after training. It costs under 1% of the original pretraining budget and still beats existing long-context methods.
Based on reporting by Sakana AI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
There's a quiet assumption baked into most Transformers: they need positional embeddings, something like RoPE, to know where each token sits in a sequence. Sakana AI's new work, DroPE, pokes at that assumption and finds it's only half true. Positional embeddings turn out to be essential for getting a model to converge during training, but the same embeddings become the thing that stops it from handling sequences longer than what it saw during training.
The fix the team lands on is almost cheeky in its simplicity. Keep the positional embeddings around while pretraining, since that's when they're doing real work, and then just drop them once the model is deployed and needs to handle longer inputs. Models trained without positional embeddings from the start, referred to as NoPE, actually generalize better to unseen lengths, but they're notoriously hard to train from scratch. DroPE sidesteps that problem entirely by letting RoPE do the heavy lifting during pretraining and then getting out of the way.
This matters for the kind of workloads that keep breaking standard pretrained models: reviewing sprawling code diffs, chewing through legal contracts, anything where the useful context blows past what the model was originally built to handle. Sakana AI tested DroPE across a range of off-the-shelf open-source LLMs and reports that recalibrating a model this way takes less than 1% of its original pretraining budget. That's a strikingly small ask for something that lets a model extrapolate to lengths it was never trained on, with no expensive long-context fine-tuning run required.
The results, according to the team, hold up against established long-context methods on benchmarks like LongBench and RULER. Sakana AI also backs the approach with a theoretical explanation for why it works: NoPE transformers trained from scratch struggle with vanishing gradients, while standard RoPE scaling techniques distort semantic attention by squeezing down low frequencies. DroPE claims to dodge both failure modes by using RoPE only long enough to get the model trained, then removing it before that distortion sets in.
The code and paper are public, which is Sakana AI's way of inviting other researchers to poke holes in the idea or build on it. Given how much compute the field pours into long-context fine-tuning, a method that gets there for a fraction of the cost is the kind of claim people will want to stress-test fast.
My take — AI-written commentary, not fact-checked reporting
Treating positional embeddings as scaffolding rather than permanent infrastructure is the kind of reframing that should embarrass anyone who assumed architecture choices made three years ago were settled science. If a sub-1% compute recalibration really holds up against dedicated long-context training on benchmarks like LongBench and RULER, every lab currently burning budget on long-context fine-tuning runs has some explaining to do. The open release is the right call too — a claim this cheap and this counterintuitive needs outside eyes on it before anyone gets too excited.
Read more about this at: Sakana AI
