Reduce RAG costs on Amazon Bedrock with query-aware compression
Amazon Web Services Aakanksha Veesam
AWS says it can cut RAG costs by shrinking retrieved text before the main model sees it. The trick kept quality close while trimming tokens and even lowered hallucinations.
Based on reporting by Amazon Web Services, Aakanksha Veesam — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS is pitching a practical way to make retrieval-augmented generation cheaper: compress the retrieved context before the main model gets it. The company’s idea is simple enough. A smaller model looks at the query and the retrieved chunks, then passes along only the verbatim spans that matter. The bigger model answers from that trimmed evidence set.
That matters because RAG systems often send a lot of text upstream. AWS says common top-k retrieval settings can pull back 5 to 20 chunks, which in technical and legal workloads can add up to several thousand input tokens per query. Those tokens are not free, and at scale they become a real part of the bill. The whole point of query-aware compression is to reduce that input load without gutting answer quality.
The pattern AWS describes fits neatly into Amazon Bedrock. A retriever, including Bedrock Knowledge Bases, returns the chunks. A single AWS Lambda function then makes two Bedrock Converse API calls: first to a smaller model, Claude Haiku in the post, and then to the primary model, Claude Sonnet. The compression prompt is unusually strict. It tells the smaller model to extract evidence, copy spans verbatim, preserve chunk IDs, and avoid paraphrasing or commentary.
AWS tested the approach on more than 500,000 documents across 9 enterprise source types and 500 questions across 10 categories. On that benchmark, compression brought cost down to 67 percent of baseline and tokens sent to the model down to 12 percent. Rerank plus compression did a little better on both, at 64 percent of baseline cost and 10 percent of tokens. Quality held close: the composite score landed at 97.5 percent for compression and 97.6 percent for rerank plus compression, versus 100 percent for baseline.
There is a tradeoff, of course. Latency rises because you’ve added another model call, with compression shown at 19 percent slower and rerank plus compression at 12 percent slower. But AWS also says the extra filtering reduced hallucinations, and the benchmark backs that up: 51 percent for baseline, 44 percent with compression, 38 percent with rerank plus compression. The company also points out that this pattern can stack with prompt caching, Intelligent Prompt Routing, and the Rerank API. Which is a very AWS move: make the pipeline longer, then sell the tools to tidy it back up.
My take — AI-written commentary, not fact-checked reporting
This is the kind of boring architecture work that actually moves the bill, which is why it matters more than another flashy demo. The real tell is that the smaller model is being asked to behave like a careful editor, not a chatty assistant. That’s a healthier pattern for RAG than shoveling more prompt into the fire and hoping for wisdom.
Read more about this at: Amazon Web Services