How we built a persistent agent memory layer on Elasticsearch with 0.89 recall and zero tenant leaks

Most AI assistants have the memory of a goldfish with amnesia. Ask a chatbot about the smart-hub reset you tried in March, then again last week, and it'll cheerfully suggest the same fix a third time. That's the problem this project set out to fix, and the solution isn't just "stuff more text into the context window." A 1M-token window is a scratchpad, not a memory system, and the well-documented lost-in-the-middle effect means facts buried mid-prompt get ignored anyway.

The architecture borrows straight from cognitive science: episodic memory for raw timestamped events, semantic memory for distilled stable facts, and procedural memory for step-by-step playbooks that track success and failure counts. Each gets its own Elasticsearch index because each has wildly different write patterns and aging rules. Episodic logs pile up fast and decay quickly. Semantic facts get curated and deduped. Procedural playbooks accumulate outcome feedback over time. Cramming all three into one bucket, the author argues, just builds a bigger haystack.

Recall runs as hybrid search: BM25 for literal matches like version numbers and error codes, dense Jina v5 vectors for semantic paraphrases, fused with RRF, then reranked by a Jina v2 cross-encoder. The system over-fetches 80 candidates per leg before reranking, because a reranker can only reorder what it's shown. Every user turn also triggers an automatic pre-recall on the raw message, since agents tend to paraphrase queries before calling tools, which strips out exactly the literal details BM25 needs.

The trickiest design problem is contradiction. When a user says they moved from Bristol to Edinburgh, the system doesn't delete the old fact, it supersedes it, tagging the new entry and hiding the old one from normal recall while keeping it queryable for audit trails. A distinction is drawn between a natural update and someone flatly denying a prior fact, with the latter written at reduced confidence until reinforced. Writes happen synchronously on every turn rather than batched at session end, because same-turn patterns, like mentioning a new device and immediately asking for a device list, need the fact visible to a tool call within the same message.

Across a 168-question eval, the system hit 0.89 recall at ten results with no cross-tenant leakage, enforced through per-user document-level security baked into Elasticsearch itself rather than a bolted-on auth layer. That last part matters more than the recall number: keeping vector search, keyword search, audit history, and access control inside one engine means fewer moving parts that can silently fail, and fewer round-trips per query.