Introducing Nested Learning: A new ML paradigm for continual learning

Google Research thinks it's found a crack in one of deep learning's oldest walls: the fact that once you train a model, its brain is basically frozen. Feed it something new afterward and it either shrugs (limited to whatever fits in the context window) or overwrites what it already knew, a failure mode researchers call catastrophic forgetting. Humans don't have this problem, at least not this badly. Your brain rewires itself constantly without erasing your childhood memories every time you learn a new recipe. Google's new paper, presented at NeurIPS 2025, argues that the reason LLMs can't do this comes down to a conceptual mistake we've been making for years.

That mistake, according to authors Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni, is treating a model's architecture and its training algorithm as two separate things. The paper, called "Nested Learning: The Illusion of Deep Learning Architectures," proposes instead that both are really just optimization problems running at different speeds, nested inside one another like Russian dolls. Some parts of the system update every step, some update rarely, and that update frequency is what defines the levels. Even backpropagation itself, they argue, can be reframed as a form of associative memory, one that's mapping data points to how surprising they were rather than doing something fundamentally distinct from what attention mechanisms already do.

Once you accept that framing, a few practical upgrades fall out of it. Google reworked standard momentum-based optimizers using an L2 regression objective instead of plain dot-product similarity, making them noticeably more robust to messy data. They also proposed something they call a continuum memory system, which stretches the usual two-tier setup in transformers (short-term attention, long-term feedforward layers) into a whole spectrum of memory modules, each refreshing at its own frequency, rather than just two fixed speeds.

To prove this isn't just theory, the team built Hope, a self-modifying variant of their earlier Titans architecture. Titans already prioritized memories based on how surprising they were, but were capped at a fairly shallow two-level learning setup. Hope removes that ceiling, letting the model optimize its own memory recursively, with the continuum memory blocks bolted on to handle longer context windows. In benchmark tests, Hope beat modern recurrent models and standard transformers on language modeling perplexity and reasoning accuracy, and it held up notably better on needle-in-a-haystack tasks designed to stress-test long-context recall.

Google is careful to frame this as early-stage groundwork rather than a finished product, and there's no indication Hope is heading into a shipping model anytime soon. But the pitch is bigger than one architecture: if you can genuinely unify the training rule and the network structure into a single system of nested optimizations, you get a new axis for designing models that isn't just "add more layers" or "add more parameters." That's the kind of idea that either quietly reshapes how the next generation of models gets built, or gets cited in a hundred papers and never leaves the lab.