Extending the Context Length to 1M Tokens!

Qwen · 1 year ago

Alibaba's Qwen2.5-Turbo extends context length from 128k to 1 million tokens, enabling processing of approximately 10 novels or 150 hours of speech transcripts. The model achieves 93.1 on the RULER long-context benchmark, surpassing GPT-4's 91.6, and reduces time-to-first-token from 4.9 minutes to 68 seconds using sparse attention mechanisms. The pricing remains ¥0.3 per 1 million tokens, allowing the model to process 3.6 times more tokens than GPT-4o-mini at equivalent cost.