Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding

Together AI · 2 months ago

Distribution-aware speculative decoding (DAS) reduces rollout time in RL post-training by up to 50% through an adaptive suffix tree drafter and length-aware scheduling that mitigates the long-tail problem where slow generations block entire training batches. Testing on math reasoning with DeepSeek-R1-Distill-Qwen-7B achieved over 50% rollout speedup while maintaining identical reward curves, and on code generation with Qwen3-8B achieved approximately 25% speedup. DAS eliminates GPU idle time during training without changing model outputs or requiring separate neural drafters, allowing practitioners to reduce compute costs while preserving model quality across varying sequence lengths and batch sizes.