Titans + MIRAS: Helping AI have long-term memory

Transformers made attention the default way for AI models to look back at what they've already read, but that lookback gets brutally expensive as sequences grow. Linear RNNs and state-space models like Mamba-2 solved the speed problem by squeezing context into a fixed-size blob, but that blob simply can't hold everything a truly long document throws at it. Google Research's answer, detailed in two new papers, is a pair of ideas called Titans and MIRAS that try to give models something closer to real memory instead of a shrinking notebook.

Titans is the actual architecture. Instead of a static matrix for memory, it swaps in a deep neural network, essentially a multi-layer perceptron, that keeps learning and updating its own weights while the model is running, with no separate retraining step required. The interesting part is how it decides what's worth remembering. Borrowing a page from human psychology, Titans uses a 'surprise metric': boring, expected tokens barely register, but anything that breaks the pattern, a banana peel photo dropped into a financial report, say, triggers a strong gradient signal that gets prioritized for long-term storage. Google also built in momentum, so useful information right after a surprising event still gets captured, and an adaptive forgetting mechanism so the memory doesn't overflow on massive inputs.

MIRAS is the theory behind it, and it's arguably the more ambitious claim. Google frames practically every sequence model, transformers included, as some version of an associative memory problem, defined by four choices: how memory is structured, what the model prioritizes learning, how it decides what to forget, and what algorithm updates it all. Most existing models lean on mean squared error or dot-product similarity, which makes them jumpy around outliers. MIRAS opens up alternatives, and Google used it to build three variants: YAAD, which uses a gentler Huber loss to shrug off one-off typos or noisy data; MONETA, which tests stricter mathematical penalties for both attention and forgetting; and MEMORA, which forces memory updates to behave like a controlled probability distribution for maximum stability.

The benchmark numbers back up the pitch. Across C4, WikiText, HellaSwag and PIQA, Titans and the MIRAS variants beat Transformer++, Mamba-2 and Gated DeltaNet at comparable sizes, while keeping the fast, parallelizable training that made RNNs attractive in the first place. Ablations showed deeper memory modules consistently produce lower perplexity and scale better as sequences stretch out. But the standout result is BABILong, a benchmark demanding reasoning across facts scattered through massive documents: Titans beat every baseline tested, including GPT-4, despite using far fewer parameters, and scaled cleanly to context windows past 2 million tokens.

Google also ran Titans on DNA sequence modeling and time-series forecasting, which suggests this isn't just a language-modeling trick. Whether any of this ships into Gemini or stays a research artifact is unclear, but the framing is bold: a unification of RNNs, transformers, and online optimization into one theoretical lens, aimed squarely at the long-context era everyone in AI keeps promising is coming.