Reading today's open-closed performance gap

Interconnects · 2 months ago

Open and closed AI models have different performance gaps depending on which benchmarks and real-world tasks are being measured, rather than a single measurable distance between them. The industry shifts its focus every 12 to 18 months—moving from chat and math capabilities to coding tasks to specialized domain work in accounting and law—making benchmark relevance constantly change. Frontier labs like OpenAI and Anthropic must continuously develop new valuable use cases to justify their infrastructure investments, while open-source models struggle most in specialized domains requiring private training data and complex evaluation environments.