Sakana AI Researchers Introduce PC-ALM, a Layer-Local Alternative to Backpropagation That Trains 1000-Layer Networks

Backpropagation still sets the standard for training neural nets, but it does everything in lockstep: forward pass, backward pass, update. Sakana AI researchers are pushing a different idea with PC-ALM, a predictive-coding variant that keeps the learning signal local to each layer instead of relying on a network-wide backward sweep.

The hook is simple enough. Standard predictive coding already replaces backprop’s global credit assignment with local prediction errors, but it struggles when networks get deep and narrow. The paper says the signal can fade before it reaches the input in those cases. PC-ALM adds a Lagrange multiplier to each layer’s constraint, so every layer gets both a prediction error and an accumulating dual variable. In the authors’ framing, that behaves like a PI controller: one term reacts to the current error, the other integrates it over time.

That extra piece seems to matter. In linear networks, the researchers prove the multipliers converge to the exact backprop adjoints at the constrained optimum, under a spectral-radius stability condition. In practice, they report that when the regular PC setup falls apart in deep, narrow models, PC-ALM stays much closer to backprop-aligned gradients. The paper’s update rule remains layer-local throughout; only the interpretation changes.

The experimental results are the real sell. On residual MLPs with widths and depths from 8 to 128, trained for one epoch on Fashion-MNIST and MNIST, PC-ALM matched backprop across the grid when given an inference budget of T = 2L. In the reference case of width 32, depth 32, ReLU on Fashion-MNIST, the repo reports 78.66% test accuracy for backprop, 68.13% for PC, and 77.75% for PC-ALM, with gradient cosine to backprop rising from 0.604 to 0.909.

And the method doesn’t stop there. The team says 1000-layer residual MLPs on MNIST, with width 32, ReLU, and 5 epochs, stay within about 2 points of backprop. They also report gains over plain PC on ResNet-18 for CIFAR-10 and Tiny ImageNet. The code is MIT-licensed, written in JAX, and runs on CPU, which makes this a research result that other people can actually poke at instead of just admire from a distance.