DiffusionBlocks: Training Neural Networks One Block at a Time
Sakana AI ● Covered by 2 sources
Sakana AI split neural network training into independent blocks instead of one giant end-to-end pass. That cuts training memory dramatically without losing accuracy across ViTs, DiTs, and LLMs.
Based on reporting by Sakana AI — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Backpropagation has been the load-bearing wall of deep learning for more than a decade. Every layer, every parameter, all held in memory at once so gradients can flow backward through the whole stack. That's fine until your network gets deep enough that the memory bill becomes the actual bottleneck, which is exactly where large models have landed. Sakana AI's new paper, DiffusionBlocks, accepted at ICLR 2026, attacks that assumption directly by asking whether you really need the whole network in memory to train it.
The answer they land on is no, and the trick is a clever reframing rather than a new architecture. Sakana treats the forward pass of a network like the denoising steps of a diffusion model. Each block's job becomes simple and local: nudge the representation a bit closer to the target than the previous block left it. Because that's precisely what a diffusion step already does, each block can be optimized on its own, with its own objective, without needing gradients from every other block in the chain.
That local objective is what lets training happen one block at a time instead of all at once. Only a single block needs to sit in memory during its own training step, not the entire depth of the network. Sakana ran this across five different architectures — Vision Transformers, Diffusion Transformers, and language models among them — and reports performance that matches standard end-to-end training, just at a fraction of the memory cost.
The idea also solves an annoying problem in recurrent-depth, or Looped, transformers, which apply the same block repeatedly and normally need backpropagation through time to train, an expensive and memory-hungry process. Under the DiffusionBlocks lens, those repeated iterations collapse into a single forward pass during training, sidestepping BPTT entirely.
What's notable here isn't a marginal efficiency tweak. It's a structural rethink of what training even requires, borrowed from a completely different modeling paradigm and grafted onto standard deep learning. If block-wise training holds up outside curated benchmarks, it changes who can afford to train deep networks in the first place.
My take — AI-written commentary, not fact-checked reporting
I like this a lot more than the usual 'we made attention cheaper by 12%' papers, because it questions a load-bearing assumption instead of shaving off overhead. Memory, not compute, is the wall most labs actually hit, and anything that decouples training cost from network depth is worth taking seriously — especially for anyone outside the handful of companies with unlimited GPU budgets. I'd still want to see this stress-tested on models bigger than the ones in the paper before calling backprop obsolete, but this is exactly the kind of foundational rethink that keeps training accessible to more than three labs on Earth.
Read more about this at: Sakana AI