Data Machina #251

Substack · 2 years ago · 14

Three new open-source language models were released: Snowflake Arctic 17B using a Dense-MoE Hybrid Transformer architecture, Apple OpenELM with four size variants from 270M to 3B parameters, and Microsoft Phi-3 models including a 3.8B instruction-tuned variant with 128K context window. Snowflake Arctic outperforms other open models in SQL conversion, coding, and RAG tasks, while OpenELM achieves comparable performance to OLMo using 2× fewer pre-training tokens through layer-wise scaling, and Phi-3 demonstrates high capability in small models deployable on devices. These releases reflect growing momentum in open-source AI with improved training techniques and data quality strategies replacing dependence on closed commercial models.