Exploring Speculative Decoding in vLLM on AMD GPUs

vLLM is taking a close look at speculative decoding, the trick that lets a model propose several future tokens and then verify them in one pass. In plain terms, that can mean fewer slow, one-token-at-a-time decode steps. But the results are messy rather than magical.

The basic idea is simple enough. A draft component suggests candidates. The target model still makes the final call. If the target accepts several draft tokens, vLLM can commit more than one output token after a single verification pass. If it rejects one, later draft tokens from that round get dropped and generation moves on from the target model’s choice.

The post compares five drafting approaches: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. They don’t all work the same way. Some generate candidates sequentially. Some work in parallel. Some pull hidden states from different target layers, while others reuse the target model’s KV cache or combine multiple target-model representations. That design choice turns out to matter.

The measurements were run on AMD Instinct MI300X and MI355X GPUs using the ROCm open software platform. And the headline is not a single winner. The effect on output-token throughput changed with the drafting method, the proposal length, the model family, the draft checkpoint, the workload, and how many proposed tokens were actually accepted.

That’s the real lesson here: speculative decoding is a systems knob, not a free lunch. The same technique that speeds up one setup can be less impressive in another, especially once the draft gets longer and the extra sequential work starts piling up before verification.