39 Lessons on Building ML Systems, Scaling, Execution, and More

Eugene Yan · 1 year ago

A practitioner shared 39 lessons learned from attending ML conferences in 2024, covering building effective ML systems, production scaling, team execution, and customer-focused product development. Key concrete points include that reward function engineering is half the battle for ML systems, LLM costs drop two orders of magnitude in 18 months, and each 10x increase in scale uncovers new operational issues. The lessons emphasize investing in robust evaluation frameworks, starting simple before adding complexity, and balancing big vision with careful attention to implementation details across cross-functional teams.