Qwen2.5-1M: Deploy Your Own Qwen with Context Length up to 1M Tokens

Alibaba's Qwen team has pushed its open-weight models into context-length territory that used to be the exclusive domain of closed frontier labs. Two months after giving the proprietary Qwen2.5-Turbo a million-token window, the team has now done the same for open-source checkpoints: Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M. Anyone can download these, run them locally, and feed them documents long enough to make most RAG pipelines feel unnecessary.

The numbers back up the claim rather than just gesturing at it. On passkey retrieval — the classic needle-in-a-haystack test — both models find a hidden string buried inside a million tokens with near-perfect accuracy, the 7B version showing only minor slippage. On tougher benchmarks like RULER and LV-Eval, the 1M versions beat their own 128K siblings by wide margins past the 64K mark, and the 14B model edges out GPT-4o-mini across multiple long-context datasets. Crucially, none of this comes at the cost of short-text performance: both models score close to their 128K counterparts on standard academic benchmarks, so you're not trading everyday usefulness for the long-context party trick.

Getting there required a layered training recipe. Qwen started from a 4K-context checkpoint, stretched it to 256K during pretraining by cranking the RoPE base frequency from 10,000 up to 10 million, then fine-tuned in two phases — short instructions first, then a mix of short and long — before a final reinforcement learning pass on shorter sequences that still generalized upward. To bridge the last gap from 256K to a full million, they leaned on Dual Chunk Attention, a technique that remaps position distances so the model doesn't choke on relative positions it never saw during training. It's a clever bit of engineering: no additional training required, and even a model trained on just 32K tokens hits near-perfect passkey accuracy at 1M.

The other half of the story is speed, because a model that can technically read a million tokens is useless if it takes forever to do so. Qwen built a sparse attention framework on top of vLLM, borrowing from MInference, that cuts prefill time by 3.2x to 6.7x depending on model size and GPU. They also tackled the memory bomb hiding in long-context inference — processing a million tokens naively burns 71GB of VRAM in the 7B model just for MLP activations — by integrating chunked prefill, which slashes that overhead by 96.7%. Even so, deploying the 14B model at full 1M-token length still demands roughly 320GB of VRAM across GPUs, so this isn't exactly a laptop hobby.

Qwen has open-sourced the full inference stack, a custom vLLM branch, and a technical report detailing the ablations, which is more transparency than most labs offer when they hit a headline number. The team is candid that long-context models still have real room to improve, and says the next push is toward more efficient architectures that don't require a small data center to run.