Speculative cascades — A hybrid approach for smarter, faster LLM inference

Google Research · 10 months ago

Researchers introduced speculative cascades, a hybrid technique combining cascades and speculative decoding to optimize large language model inference by using a smaller draft model whose output a larger model verifies in parallel with flexible deferral rules. Testing on Gemma and T5 models across summarization, translation, reasoning, coding, and question-answering tasks showed speculative cascades achieved better cost-quality trade-offs than either baseline technique alone. This approach enables developers to control the balance between inference speed and output quality without sacrificing one for the other.