Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction

Google Research just published details on a clever fix for a problem anyone who's used on-device AI features has probably felt without knowing it: that slight lag before your phone finishes summarizing notifications or proofreading a text. The culprit is autoregressive generation, the standard way language models work, spitting out one token at a time. On a server farm that's fine. On a Pixel with a fixed energy budget and limited RAM, it's a bottleneck that chews through battery and slows everything down.

The fix is called Multi-Token Prediction, or MTP, and Google's approach here is notably pragmatic. Rather than train a whole new drafting model to guess ahead — the usual speculative decoding trick, where a small model proposes tokens and a big model verifies them — Google took its already-deployed Gemini Nano v3 model, froze its weights entirely, and bolted on a lightweight transformer head trained to predict several tokens ahead using the main model's own internal activations. No retraining the backbone, no risk of messing with safety alignment or capability, and because rejected drafts just get discarded, output stays bit-for-bit identical to what the unmodified model would have produced.

The more interesting engineering choice is what Google calls a zero-copy architecture. Standalone drafter models typically have to build and maintain their own memory cache as they process context, which on a phone means paying a "double tax" in RAM. Google's MTP head instead cross-attends directly into the main model's existing key-value cache, borrowing context instead of duplicating it. That saved roughly 130MB per instance compared to a standalone drafter of similar size — a meaningful chunk of memory on a device that's also running your camera app and everything else.

The payoff, according to Google, is speedups of 50% or more on Pixel 9 for tasks like AI Notification Summaries and Proofread, now rolled out to Pixel 9 and 10. Because the drafter head can see the main model's rich internal state rather than treating it as a black box, it's noticeably better at guessing what comes next, especially on structurally predictable text like smart replies, where token acceptance improved by up to 55%. In production, that translates to nearly two extra correctly-predicted tokens per inference pass, which means fewer expensive verification cycles and less time keeping power-hungry chips awake.

Google says it's not stopping here, with plans to explore parallel decoding, branching prediction paths for ambiguous language, and looser verification rules that trade a bit of strictness for more speed on specific tasks. It's a small architectural tweak with an outsized practical effect, and it's a good example of how a lot of real-world AI progress right now isn't about bigger models but about squeezing more out of the ones already sitting in your pocket.