Five AI agent harnesses took the same memory test. The results weren’t even close.
The New Stack Amanda Caswell
Five AI agent systems got the same memory test, and the scores were all over the place. The harness mattered a lot: the best one beat the weakest by a wide margin.
Based on reporting by The New Stack, Amanda Caswell — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Letta just put a number on a hunch many developers already had: memory alone doesn’t make an agent better at work. Its new FrontierMemory benchmark, released Wednesday, tries to measure whether systems can actually learn from past tasks instead of merely repeating back stored facts. The early results are messy in a useful way. Five agent harnesses — the software that manages how an agent operates — produced very different scores on the same test.
Letta Dream led the pack with 73.3%. Prime Agent followed at 60%. OpenAI Agents SDK landed at 40%, while Claude Managed Agents and Hermes Agent finished at 33.3% and 26.7%. And the gap wasn’t just about the model inside the system. When Letta ran Dream and OpenAI Agents SDK with the same models for memory generation and task completion, Dream still came out ahead, 60% to 40%.
That’s the point Charles Packer is pushing. Recall benchmarks have gotten too easy, he argues, because they mostly test whether an agent can retrieve facts. What still looks unsolved is continual learning in token space: using the context an agent keeps, not weight updates, to improve the next task. Sarah Wooders frames it more like onboarding. A new hire needs step-by-step direction; someone who’s been around for years can often get moving with less help.
FrontierMemory is built around that idea. Agents get transcripts of earlier work, including both successes and failures, then write their own memories to help with later tasks. Letta rewrote existing coding and knowledge-work tests so the agents had to use what they had learned before, then narrowed the first version down to 15 tasks after fixing issues in the original setup. The benchmark is still small, and Letta says one task can move scores by roughly 6.7 percentage points.
Letta also tested seven frontier models on memory generation. Claude Opus 5.5 and Z.ai’s GLM-5.3 tied for first at 73.3%, followed by Kimi K3 at 66.7%. OpenAI’s GPT-6.1 Sol and Google’s Gemini 4 Argon both scored 60%, while Muse Wonder and Grok 4.7 came in lower. GPT-6.1 Sol was cheaper, though: Letta estimated it at $0.77 per task, versus $2.80 for Opus 5.5 and $3.58 for GLM-5.3. That makes the benchmark’s awkward little message hard to ignore: the best memory may not come from the cheapest model, but the cheapest model may still be the one you want doing the daily grind.
My take — AI-written commentary, not fact-checked reporting
This is the part of agent hype that keeps getting skipped over: the harness is not a neutral wrapper, it’s half the product. If a better memory system can rescue a weaker model, then the industry’s obsession with raw model scores is already looking a bit silly. Also, “memory” that remembers the wrong thing is just bureaucracy with better branding.
Read more about this at: The New Stack