Latest open artifacts (#21): Open model bonanza! Gemma 4, DeepSeek V4, Kimi K2.6, MiMo 2.5, GLM-5.1 & others. On CAISI's V4 assessment.

Interconnects · 2 months ago

Multiple open-source AI labs released new frontier models this month, including DeepSeek V4, Gemma 4, Kimi K2.6, and others, prompting a fresh evaluation of how open models compare to closed alternatives. The Center for AI Standards and Innovation found open models lag by 3-7 months behind American frontier models, with the gap widening when measured against proprietary benchmarks like CTF-Archive-Diamond and PortBench. The benchmarking methodology itself remains contested, as standardized evaluation setups do not match how models are actually deployed—for example, using basic coding harnesses rather than the specialized environments models are trained for.