Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding

Reinforcement learning post-training has a dirty little secret: most of the clock time isn't spent updating weights, it's spent waiting. Together AI's research points out that rollout generation — the step where a model writes out full responses before training can proceed — eats up roughly 70% of total training time in RL pipelines, more than backprop and optimizer updates combined. The culprit is a classic long-tail problem. Most responses finish quickly, but a handful of long chain-of-thought generations drag on, and because training steps are synchronous, the whole batch sits frozen until the slowest one finishes. GPUs that could be crunching numbers instead burn cash idling.

Together's fix, called distribution-aware speculative decoding (DAS), attacks this without touching what the model actually outputs. The core trick is an adaptive suffix tree drafter, a nonparametric, training-free predictor built from a sliding window of the model's own recent rollouts. Because RL training revisits the same prompts across epochs, there's a lot of repeated structure to exploit — DAS matches the current generation prefix against this history, predicts likely next tokens based on frequency, and lets the target model verify the guesses in parallel. Verified tokens get folded straight back into the tree, so the drafter tracks the policy as it evolves after every optimizer step, with no retraining and no accumulating memory overhead across a long training run.

The second half of the system is scheduling. DAS spreads long-running requests across GPUs instead of letting them pile up on one worker, and it starts applying speculative decoding to long requests early, before they become late-stage stragglers stuck in slow, memory-bound small-batch decoding. Within a single GPU, requests get sorted into long, medium, and short buckets based on historical rollout stats, with speculation budgets allocated accordingly — short requests skip speculation entirely since there's nothing to gain there.chip time isn't wasted guessing tokens that were never going to be a bottleneck.

On math reasoning with DeepSeek-R1-Distill-Qwen-7B, DAS cut rollout time by more than 50% on a 1,209-example dataset while reproducing the exact same reward curve as standard decoding. On code generation RL with Qwen3-8B using unit-test rewards, the gain was closer to 25%, again with reward quality tracking the baseline throughout. The speedups held steady across sequence lengths from 8k to 16k tokens and batch sizes from 16 to 32, which suggests this isn't a narrow benchmark trick tuned to one configuration.

What's notable here isn't just the raw speedup — it's that DAS is lossless. Outputs and training curves match the unmodified baseline exactly, so there's no quality tax for the compute savings. As reasoning models keep generating longer chains of thought and RL post-training keeps scaling up, that rollout bottleneck was only going to get worse. A method that reclaims half the wasted GPU time for free, with zero maintenance overhead, is the kind of unglamorous infrastructure work that ends up mattering more than another leaderboard score.