Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs
Hugging Face 1 week ago 40 ● 3 sources
Olmo released Olmo-core 3, an open training framework with a redesigned mixture-of-experts system for scaling large language-model training. In one benchmark, it increased the expert pool from 8 to 128 while keeping about 3.2B active parameters per token and limiting throughput drop to under 5%. It replaces an earlier FSDP-based MoE approach with a DDP-style expert-resident training stack (and other routing/precision optimizations) to improve throughput and enable scaling up to trillion-parameter MoE configurations.