Speculative Decoding for 2x Faster Whisper Inference

Hugging Face · 20 December 2023, 00:00

Researchers demonstrated that speculative decoding reduces Whisper speech transcription inference time by a factor of 2 while producing identical outputs. The method uses a faster assistant model to generate candidate tokens that are verified by the main model in a single forward pass, reducing inference time from 73 seconds to 33 seconds on a test dataset. This makes speculative decoding a plug-in replacement for existing Whisper pipelines without sacrificing transcription accuracy.