Sakana AI’s Error Diffusion Trains Dale-Compliant Dual-Stream Networks, Reaching 96.7% MNIST and 61.7% CIFAR-10 Without Backpropagation

Backprop has a dirty secret: it needs the exact mirror image of forward weights to send errors backward. Real neurons can't do that trick, which is why neuroscientists have spent decades hunting for training methods that don't require this so-called weight transport. Sakana AI's new paper, Diffusing Blame, revives an old and mostly forgotten idea called Error Diffusion, first proposed by Kaneko back in 2000, and pushes it far past where it had ever gone before.

The trick is architectural. Every layer gets split into two parallel streams, one excitatory and one inhibitory, mimicking how cortical circuits actually separate these roles. All the learnable weights stay non-negative, and the inhibitory connections are hardwired as subtractions rather than learned negative values. That's Dale's principle, baked directly into the math instead of hoped for. The catch is it roughly quadruples the parameter count compared to a single-stream network doing the same job.

The bigger technical unlock is something the team calls modulo error routing. Instead of using random feedback matrices like the popular Direct Feedback Alignment method, each hidden unit gets permanently assigned to one output channel via a simple modulo operation, and learns only from that channel's error. Combined with three fixes — wider sigmoids to stop error signals from decaying 25-fold by the first layer, batch-centered error to counter class imbalance, and asymmetric excitatory/inhibitory initialization — this pushed accuracy from a near-random 50.4% and 11.6% up to 96.7% on MNIST and 61.7% on CIFAR-10. That's the first time this rule has trained a convolutional network at all.

What's genuinely interesting is that the three fixes matter in completely opposite ways depending on the dataset. Drop the wide sigmoids and MNIST accuracy craters by 71 points while CIFAR-10 barely notices. Drop the batch-centering and it's the reverse: CIFAR-10 collapses by nearly 48 points while MNIST shrugs. That's not noise, it's a sign that different tasks bottleneck credit assignment in different places, something a single-benchmark victory lap would never reveal.

The team also plugged this into reinforcement learning, pairing it with PPO to make ED-PPO. It beat standard backprop-trained PPO on the HalfCheetah locomotion task and held its own against DFA on Ant. On the open-ended Craftax benchmark, DFA actually became the weakest method of the bunch, suggesting the random-feedback shortcut that works fine for classifying digits doesn't generalize to messier, more exploratory settings. None of this beats real backprop yet — DFA still edges out Error Diffusion by up to 7 points on the benchmarks — but for hardware that physically can't represent negative synaptic weights, like photonic or neuromorphic chips, having a biologically plausible option that's actually competitive is a real step forward.