Agentic retrieval for Amazon Bedrock Managed Knowledge Base

Ask a chatbot to compare your company's 2020 strategy to 2023, and standard retrieval-augmented generation quietly falls apart. It smashes your multi-part question into one embedding vector, searches once, and hands back five chunks that average out every sub-intent into mush. AWS's new AgenticRetrieveStream API, now live for Amazon Bedrock Managed Knowledge Bases, is built to fix exactly that failure mode.

The mechanism is refreshingly close to how a competent analyst actually works. Instead of firing one query and calling it done, a foundation model plans the search: it splits "compare hiring, investment, and customer obsession in 2020 vs 2023" into separate sub-queries, retrieves evidence for each, checks whether it has enough, and loops again if not. AWS tested this against 25 years of Amazon shareholder letters and found the old single-shot Retrieve API's top result for "most important message in the documents" was, memorably, a passage about the Amazon logo's color scheme. Technically a high similarity score. Practically useless.

What makes this more than a wrapper around GPT-style looping is the trace system. Every planning step, sub-query, and retrieval call streams back as an event you can inspect — SpeculativeRetrieval fires before planning even starts to cut latency, then Planning, Retrieval, and a final deduplicated Result event carry the receipts. Teams have been hand-rolling this exact pattern for a year or two, bolting loop logic onto the plain Retrieve API, managing their own stopping conditions and dedup. AWS is essentially productizing that duct tape.

The numbers back up the pitch, at least on paper. On MuSiQue, a public multi-hop QA benchmark, agentic retrieval posted a 20-point absolute recall improvement over single-shot search, with the gap widening as questions get harder — a 37.3-point gain on four-hop questions versus under five points on single-hop ones. It also tends to use close to the ideal number of retrieval steps a human annotator mapped out, rather than either quitting early or wandering. That's a meaningful signal: the system isn't just doing more work indiscriminately, it's roughly matching effort to actual question complexity.

None of this is free, and AWS is upfront about the tradeoff. You're paying for multiple foundation model calls per question — $4 per 1,000 agentic calls plus $1 per 1,000 underlying retrievals with the managed model, more if you bring your own — and latency goes up accordingly, since it's a streaming-only, multi-invocation loop capped at up to five iterations. For a quick lookup, plain Retrieve is still faster and cheaper. But for the genuinely gnarly comparative questions that used to require custom orchestration code and still often disappointed users, this closes a real gap in Bedrock's retrieval stack, including native routing across up to five separate knowledge bases based on plain-English descriptions.