The Sequence AI of the Week #878: Inside Google Deepmind's First Real Crack in Next-Token Generation
Substack Jesus Rodriguez
Google DeepMind built DiffusionGemma, a text model that generates by refining a whole draft at once instead of writing word by word. It's a real challenge to the token-by-token method every chatbot uses today.
Based on reporting by Substack, Jesus Rodriguez — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
For years, basically every language model you've talked to has worked the same way: one token at a time, left to right, no backtracking. GPT-4, Claude, Gemini, your coding assistant — they all commit to a word, move on, and never look back at what they just wrote. It's a strange constraint when you think about it, because humans don't write that way. We draft, we revise, we jump around a paragraph fixing things out of order. Google DeepMind's new model, DiffusionGemma, is built on the bet that machines shouldn't have to write that way either.
Instead of stamping tokens onto a page like a typewriter, DiffusionGemma starts with something closer to noise or a rough sketch of the entire output, then refines the whole thing in parallel, pass after pass, until it converges on coherent text. This is the diffusion approach that has quietly powered image generators like Stable Diffusion and Midjourney for years, now pointed at language instead of pixels. The appeal is obvious once you say it out loud: a model that can revise its own draft, rather than being locked into whatever it committed to three tokens ago.
What makes this release notable isn't just that DeepMind tried diffusion for text — plenty of research labs have poked at that idea before, usually with underwhelming results. It's that DiffusionGemma is genuinely competitive, built on the Gemma family that Google already ships as a serious open-weight line. That's the detail that turns this from an academic curiosity into something the rest of the field has to take seriously. Transformers have had an eight-year head start and an enormous amount of engineering investment behind them; a diffusion-based model closing that gap, even partially, is a signal that the autoregressive default isn't as inevitable as it looked a year ago.
There are real tradeoffs here, and DeepMind isn't pretending otherwise. Generating a full draft and iteratively denoising it is a different computational shape than predicting one token and moving on, and it changes how you think about latency, batching, and even what
My take — AI-written commentary, not fact-checked reporting
I've been skeptical of every 'transformers are dead' claim that's crossed my desk this year, and I'm still skeptical — but this one's different because it's Google shipping it, not a startup blog post chasing funding. Diffusion for text won't replace autoregressive models overnight, but it's the first credible sign that next-token prediction is a design choice, not a law of physics, and that alone should make every lab re-examine assumptions they stopped questioning years ago.
Read more about this at: Substack