Reading today's open-closed performance gap
Interconnects Nathan Lambert
Open and closed AI models have different performance gaps depending on which benchmarks and real-world tasks are being measured, rather than a single measurable distance between them. The industry shifts its focus every 12 to 18 months—moving from chat and math capabilities to coding tasks to specialized domain work in accounting and law—making benchmark relevance constantly change. Frontier labs like OpenAI and Anthropic must continuously develop new valuable use cases to justify their infrastructure investments, while open-source models struggle most in specialized domains requiring private training data and complex evaluation environments.
Why it matters
The complex factors that determine the single evaluation number so many focus on. Plus, how this changes in the future.
