Building A Generative AI Platform
Chip Huyen
The article outlines a modular architecture for deploying generative AI applications, starting from basic model queries and progressively adding components like context augmentation, retrieval-augmented generation (RAG), guardrails, routing, caching, and observability. Key retrieval approaches include term-based methods like BM25 (faster, cheaper) and embedding-based vector search (more computationally expensive but improvable), with hybrid search combining both approaches and reranking to optimize results. The platform design enables models to access external data sources, SQL tables, and web search through read-only and write actions, with query rewriting to improve retrieval accuracy in multi-turn conversations.
Why it matters
After studying how companies deploy generative AI applications, I noticed many similarities in their platforms. This post outlines the common components of a generative AI platform, what they do, and how they are implemented. I try my best to keep the architecture general, but certain applications might deviate. This is what the overall architecture looks like. This is a pretty complex system. This post will start from the simplest architecture and progressively add more components. In its simplest form, your application receives a query and sends it to the model. The model generates a response, which is returned to the user. There are no guardrails, no augmented context, and no optimization. The Model API box refers to both third-party APIs (e.g., OpenAI, Google, Anthropic) and self-hosted APIs. From this, you can add more components as needs arise. The order discussed in this post is common, though you don’t need to follow the exact same order. A component can be skipped if your syst
