The Sequence Opinion - Issue 939: Beyond the Next Token
TheSequence Jesus Rodriguez
Opinion — commentary, not a factual news event.
Text generation usually can’t rewrite itself once it speaks. Diffusion can fill in many words at once, which could make editing and generation faster.
Based on reporting by TheSequence, Jesus Rodriguez — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Writing with an autoregressive model is a bit like typing on a keyboard that only lets you append. You can pause and think, but once a token is out, the next one has to deal with it. The model can correct itself later, but it cannot quietly go back and change what it already said.
Text diffusion takes a different route. It begins with something incomplete or corrupted and then builds an answer by repeatedly denoising it. More than one position can turn into words in the same step. That opens the door to faster generation and more flexible editing.
But that flexibility comes with its own problem. If several parts of a sentence are being decided at once, those decisions still have to agree with each other. If they don’t, the model can burn away the speed advantage on extra computation just to keep the result coherent.
One thing is easy to miss here: diffusion is not the opposite of a transformer. A transformer is the network architecture. Autoregression and diffusion are ways of training and using it. That is why models like LLaDA can use transformers and still behave very differently from the usual next-token setup.
My take — AI-written commentary, not fact-checked reporting
This is the kind of distinction the AI crowd loves to wave past while chanting about “the future of generation.” The real story is simpler: next-token models are convenient, but not sacred. Anything that makes text less trapped by its own first mistake deserves a serious look, even if the hype machine has to learn a new buzzword before breakfast.
Read more about this at: TheSequence