AI Engineer 2023 Keynote - Building Blocks for LLM Systems

Eugene Yan · 2 years ago

Eugene Yan presented a keynote on design patterns for building production LLM systems at the 2023 AI Engineer Summit, covering evaluations, retrieval-augmented generation, guardrails, and feedback collection. Key concrete findings included that document position significantly affects RAG accuracy (75% at best with perfect retrieval, dropping to worse-than-baseline when answers are in middle positions), and that an eval set of 40 handcrafted questions can effectively validate domain-specific tasks. The patterns outlined—eval-driven development, improved retrieval ranking, and factual consistency checking via natural language inference—provide practical approaches to address limitations in current LLM system deployment.