Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster

Researchers from Stanford and UC Berkeley have put out TokenGate, a module designed to speed up long-context prefill in existing language models. The basic idea is simple enough: let the model skim first, then spend full attention only on the tokens that actually matter. That keeps the document part of a prompt prunable, while the question stays intact.

TokenGate bolts onto pretrained models with a small MLP gate and LoRA, set at r=16 in the reported runs. The team tested it on Qwen3-1.7B and Qwen3.5-2B with contexts as long as 128k tokens. On paper, the setup ran on a single A100 at batch size 1 for timing, with training spread across four A100s.

The numbers are the hook. Across five tasks, TokenGate delivered 1.4x to 11.3x faster time-to-first-token versus full prefill, while peak prefill memory fell by 1.5x to 3.4x. The best result came on NIAH 16k, where TTFT hit 11.3x faster and memory dropped 3.42x. At 128k, the speedup reached 13.0x. The floor was lower too: on some tasks, the gain shrank to 1.4x.

This is not just a crude pruning trick. The selector is trained, not bolted on as a fixed heuristic. TokenGate borrows a mixture-of-experts style router idea so the top-K choice can pass gradients back to the scorer. A learnable gain and temperature help tune the attention bias above the gate. Training uses a plain next-token loss, mixing a full-context objective with a random-budget one so the same gate can handle different budgets.

Against FastKV, SlimInfer, DASH, GemFilter, FlexPrefill and MInference, the system was Pareto-optimal on the five-task suite. Compared with the next-best baseline, it was 1.2x to 6.4x faster on TTFT and used up to 2.1x less memory. The team also says TokenGate works on Qwen3.5-2B, which mixes Gated DeltaNet and full-attention layers, where FastKV cannot prune the DeltaNet side at all.