Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
Sakana AI ● Covered by 2 sources
Sakana AI introduced DroPE, a method that extends the context length of pretrained large language models by removing positional embeddings after training, eliminating the need for expensive fine-tuning. The approach requires less than 1% of the original pretraining compute budget while outperforming established methods on benchmarks like LongBench and RULER. This allows models to handle longer sequences in real-world applications such as reviewing code diffs or analyzing legal documents without additional computational overhead.
