Aurora

Speculative decoding has always had an expiration date problem. You train a small draft model to guess ahead of the big target model, ship it, and within weeks the traffic mix shifts, the target model gets swapped for a safety or cost reason, and the draft's guesses start getting rejected more often. Offline retraining exists, but it's slow, and at production scale the activation-collection pipelines needed to feed it can balloon into petabytes of storage. Together AI's answer, released today as an open-source framework called Aurora, is to stop treating the draft model as something you train once and ship, and instead let it keep learning from the exact requests it's serving.

The mechanics are split into two loosely coupled pieces. An inference server, built on SGLang or vLLM, runs the usual speculative decoding loop — draft model proposes tokens, target model verifies them in parallel — and streams both the accepted and rejected outcomes into a shared data buffer. A separate training server pulls batches from that buffer, updates a copy of the draft model, and periodically swaps the improved weights back into serving without taking anything offline. Together frames this as reinforcement learning rather than imitation: the draft model is the policy, the verifier is the environment, accepted tokens are reward, and rejected proposals become counterfactual signal the model learns to avoid. A custom tree-attention mask lets Aurora process all the accepted and rejected branches of the speculative tree in one batched pass, which is what makes the whole thing cheap enough to run continuously.

The numbers are the interesting part. Aurora was tested against a well-trained static speculator across Qwen3 and Llama3 models and delivered an extra 1.25x speedup on top of that already-tuned baseline — gains that came purely from continuing to adapt online. In a stress test simulating 40,000 prompts across five domains (math, text-to-SQL, code, finance, general chat), fed in abrupt domain-by-domain bursts, Aurora recovered its acceptance length within roughly 10,000 requests every time the traffic shifted. A static speculator has no mechanism to do that at all; it just degrades and stays degraded.

The more provocative finding is what happened with shuffled, mixed traffic. Training a speculator from scratch, live, online, beat a speculator that had been carefully pretrained offline and then fine-tuned — 3.08 average accepted tokens versus 2.99, with throughput settling at 302.3 tokens per second. That's a real dent in the assumption that speculative decoding needs heavy offline investment before it's worth deploying. On real hardware benchmarks with MiniMax M2.5 and Qwen3-Coder, Together also reported speedups up to 1.92x over no speculation at all, depending on batch size.

None of this replaces the target model or changes what speculative decoding fundamentally does. What it changes is the operating assumption: that a draft model is a fixed artifact you retrain on a schedule, rather than a live component that should keep tracking the traffic and the target model as both drift underneath it.