Reflections on AI Engineer Summit 2023

Eugene Yan · 2 years ago

An attendee at the first AI Engineer Summit in San Francisco reported that evaluation and serving costs are the top challenges for LLM deployment, based on survey responses from over 800 practitioners. The most concrete findings include that LLM-based grading for evals becomes expensive at scale, code generation is the easiest task to evaluate automatically, and using LLMs for complex work like writing plugins costs around $10 versus weeks of developer time. As a result, teams are focusing on building evals through simple assertions, periodic human review, and caching strategies, while prioritizing LLM deployment for high-complexity tasks rather than simple repetitive ones.