Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference
Amazon Web Services Kareem Syed-Mohammed
AWS added prefix-aware routing to SageMaker Inference. It keeps repeated LLM prompts on the same instance, so cached prompt work finally pays off.
Based on reporting by Amazon Web Services, Kareem Syed-Mohammed — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Amazon SageMaker Inference now has a routing trick built for LLMs that keep seeing the same opening text over and over. Instead of spraying requests across a fleet at random, prefix-aware routing looks at the start of the prompt and tries to send matching requests to the same instance. That matters because the expensive part of many prompts is the fixed block up front: instructions, retrieved documents, chat history, policy text. The user’s actual question can be tiny by comparison.
The point is to let prefix caching do its job. Frameworks like vLLM and TensorRT-LLM can reuse cached key-value pairs when the same prefix comes back, but only if the request lands on the same machine again. AWS says its new routing strategy closes that gap. In benchmarks on Llama 3.1 70B, it cut P50 time-to-first-token by up to 77 percent and lifted throughput by up to 16 percent, while KV cache hit rates rose from roughly 25 percent to over 80 percent.
The gains were bigger on long prompts, which makes sense. With 8,000-token shared prefixes running for an hour, P90 TTFT fell 33 to 37 percent and P50 TTFT fell 71 to 77 percent. Shorter ShareGPT-style conversations still improved, but less dramatically: P90 TTFT dropped 24 to 37 percent and throughput rose 1.7 to 2.0 percent. AWS also says the routing logic adds only 1.3 to 1.9 milliseconds per request, while model TTFT in the tests ranged from 63 to 280 milliseconds. In other words, the overhead barely shows up.
There are guardrails, too. If one prefix gets too hot and an instance hits the concurrency limit you set, SageMaker can overflow traffic to a less busy machine. And when you scale the fleet up or down, most requests keep going to the same place, so the cache does not get wiped every time capacity changes. AWS says traffic stayed balanced across seven ml.p5.48xlarge instances, with each one handling 13.3 to 15.4 percent of requests.
The feature is aimed squarely at workloads where the beginning of the prompt repeats: RAG systems, multi-turn chat, templated assistants, code completion. It works with single model endpoints, inference components, the native Invoke API, and the OpenAI-compatible API. You enable it at endpoint configuration time with RoutingStrategy set to PREFIX_AWARE, plus PrefixLength and ConcurrencyThreshold. No model container changes needed. The catch is the same old one: prefix caching has to be on in your serving stack, or the routing can only do so much.
My take — AI-written commentary, not fact-checked reporting
This is the kind of unglamorous infrastructure work that actually moves the bill. Everyone loves talking about bigger models; fewer people want to admit that smarter routing can save more pain than another shiny benchmark chart. Closed, open, whatever — if the prompt starts the same way every time, the machine should stop pretending it’s new.
Read more about this at: Amazon Web Services