DiffusionBlocks: Training Neural Networks One Block at a Time

Sakana AI ● Covered by 2 sources

Sakana AI introduced DiffusionBlocks, a training method that splits neural networks into blocks trained independently by treating the forward pass as a diffusion model denoising process. The approach, accepted at ICLR 2026, reduced memory requirements from linear growth with network depth to memory for a single block while matching performance on ViTs, DiTs, and LLMs. This allows training deep networks without holding the entire model in memory simultaneously, addressing a fundamental constraint in current AI training infrastructure.