Configure rate limits for AI traffic on AgentCore gateway

AWS just gave AgentCore gateway users a lever they've been asking for: proper rate limiting. AgentCore gateway is the managed front door Amazon Bedrock uses to route AI traffic to web search tools, knowledge bases, MCP servers, LLMs, and agent-to-agent calls. Until now, it routed everything but gave you no real way to stop one aggressive caller from eating the whole pipe. That changes with the new per-user, per-target, and per-token controls AWS rolled out this week.

The mechanism is built around two ideas: dimension keys and entries. Dimension keys tell the gateway how to bucket incoming requests, by target name, tool name, model ID, or claims pulled straight out of a JWT or IAM identity. Entries then say how much throughput each bucket gets. AWS walks through a scenario with three fictional user tiers, Basic, Advanced, and Beta, authenticated via Microsoft Entra ID, where Beta testers get generous limits on restricted models specifically so a company can benchmark new models before wider rollout, while Basic users get capped hard so they can't accidentally (or deliberately) flood a shared MCP tool called Booking.

Three metrics get tracked: requests per second or minute, tokens per minute for inference targets, and connections per second. The connection metric is the clever one. A request that finishes in 50 milliseconds counts the same as one that streams for 90 seconds under RPS limits, but under CPS, that 90-second stream holds a slot the entire time. That distinction matters a lot for anyone running long-lived streaming inference, since a handful of slow sessions can quietly choke a target that looks fine on a request-count graph.

AWS also spells out a subtlety that's easy to miss: group-level limits and per-user limits stack with AND logic, not OR. So even if the Basic tier collectively has capacity left, one user who's already burned through their personal 20-requests-per-minute cap gets throttled anyway. It's a two-layer defense, one ceiling stopping one group from starving another, another stopping one loud person from starving their own teammates. AWS's example with an engineer named Arnav hitting his personal cap while his group still has headroom makes the point cleanly. On top of all customer-defined rules sits a hard service-managed quota that no configuration can exceed, so there's a backstop even if someone misconfigures their own limits.