How Couchbase built a multi-model AI architecture for Capella iQ with Amazon Bedrock

Couchbase's Capella iQ is the AI sidekick that writes SQL++ queries, suggests indexes, and chats with developers about their database. Cool feature, but running it on a single hardcoded model is a recipe for pain the moment that model gets slow, expensive, or replaced. So Couchbase went and rebuilt the plumbing behind Capella iQ, and the result is a genuinely interesting case study in how to do multi-model AI infrastructure without losing your sanity.

The architecture runs on Amazon EKS across two Regions, us-east-1 and us-west-2, with three microservices doing the heavy lifting: cp-api handles incoming developer requests, cp-internal-api resolves which model vendor to route to, and cp-ns manages per-tenant model configuration. Requests never touch the public internet — everything flows through a private VPC endpoint straight to Amazon Bedrock's runtime, which matters a lot if you're selling to enterprise customers who care about SOC, HIPAA, and ISO compliance. Bedrock's Cross-Region Inference feature then spreads the actual inference load across us-east-1, us-east-2, and us-west-2 automatically, so a regional hiccup or traffic spike doesn't require anyone on Couchbase's team to manually flip a switch.

Before shipping anything, Couchbase built a benchmark suite covering the five workflows Capella iQ actually needs to nail: SQL++ generation, index recommendations, query explanations, insights generation, and multi-turn conversation. They scored candidate models on correctness, determinism, latency, and formatting consistency. Anthropic's Claude Sonnet 4.5 came out on top, hitting roughly 76 percent accuracy on an internal test modeled after the BIRD benchmark, with no critical regressions across any workflow. That number became the bar, and Claude Sonnet 4.5 became the production model — for now.

The harder part wasn't picking a model, it was proving the failover actually works. Simulating regional throttling and partial endpoint degradation isn't something you can casually reproduce in a dev environment, so the team built custom test harnesses and worked directly with AWS to tune retry and timeout settings until requests reliably rerouted to a healthy Region without users noticing a slowdown. They also had to normalize benchmark results across models with different tokenization and formatting quirks just to compare them fairly, which ate more engineering time than you'd expect.

What's next is arguably the more telling part: Couchbase wants to use Bedrock's Custom Model Import to run smaller, distilled models for high-volume, narrow tasks like index recommendations, cutting inference costs while keeping quality steady. That's the real payoff of building a provider-agnostic layer in the first place — swapping in Claude Sonnet 5 or a cheaper fine-tuned model later becomes a namespace config change, not a rewrite.