Speculative cascades — A hybrid approach for smarter, faster LLM inference

Running large language models is expensive, and the industry has spent the last couple years hunting for ways to make inference cheaper without gutting quality. Two techniques have emerged as the go-to fixes: cascades, which let a small model handle easy queries and punt hard ones to a bigger model, and speculative decoding, which uses a small "drafter" model to guess ahead while the big model verifies in parallel. Google Research's new paper, "Faster Cascades via Speculative Decoding," argues both approaches leave something on the table, and proposes a hybrid it calls speculative cascades.

The problem with plain cascades is that they're sequential. The small model has to finish its answer and self-assess confidence before anything gets handed off, so a low-confidence case means you paid for the small model's work and still have to start the large model from scratch. Speculative decoding fixes the latency problem by having both models work in parallel, but it enforces strict token-by-token matching against the large model's output. Google's own example is telling: asked "Who is Buzz Aldrin?", a small model and a large model both give correct, high-quality answers, but they phrase the first word differently — "Buzz" versus "Edwin." Under speculative decoding's rigid matching rule, that single mismatch torches the entire draft, even though the small model's answer was perfectly good.

Speculative cascades keep the parallel drafting-and-verifying structure of speculative decoding but swap out the rigid acceptance rule for what Google calls a flexible deferral rule. Instead of requiring exact token matches, the system decides on the fly, token by token, whether the small model's guess is good enough to keep or whether it's worth deferring to the big model. Google describes several versions of this rule: a simple confidence threshold, a comparative check between the two models' confidence levels, a cost-benefit calculation weighing the gain in confidence against the cost of rejecting a draft, and a check against the large model's list of top-ranked acceptable tokens. Any of these can be swapped in depending on what a developer cares about most — speed, accuracy, or some balance of the two.

Google tested the approach on Gemma and T5 models across summarization, translation, reasoning, coding, and question-answering tasks, benchmarking against standard cascades and standard speculative decoding. The headline result: for the same output quality, speculative cascades needed fewer calls to the large model, which translates directly into faster generation and lower cost. On a GSM8K math word problem, the paper shows the hybrid method reaching the correct answer faster than plain speculative decoding by accepting more of the small model's reasonable-but-not-identical tokens along the way.

None of this is a wholesale replacement for either underlying technique — it's closer to a smarter switch sitting between them. But as more products lean on multiple model sizes stitched together behind the scenes, the deferral logic Google describes here looks like exactly the kind of plumbing that will end up buried invisibly inside every serving stack, the sort of infrastructure improvement users never see but constantly benefit from.