Titans + MIRAS: Helping AI have long-term memory

Google Research · 7 months ago

Researchers introduced Titans, a neural architecture, and MIRAS, a theoretical framework, that combine fast RNN-like processing with transformer-like accuracy by using deep neural networks as adaptive memory modules that selectively store surprising information. Titans achieved lower perplexity than Mamba-2 and Transformer++ on standard benchmarks and outperformed GPT-4 on the BABILong task despite having far fewer parameters. This enables models to efficiently handle context windows larger than 2 million tokens while adapting their memory in real-time without offline retraining.