Modular LLMs at scale: how FlexOlmo is helping to pool national expertise without pooling sensitive data
Allen Institute (AI2)
Researchers at the University of Southern Denmark built FlexMoRE, a modular language model that allows separate institutions to train specialized components independently without sharing sensitive data. FlexMoRE reduces memory demands to less than one-third of the original FlexOlmo model while maintaining performance by using smaller low-rank adapters instead of full-size experts. This enables distributed model training for lower-resource languages like Danish while preserving privacy, allowing organizations with restricted data to contribute expertise to shared models.
Why it matters
Danish Foundation Models is using FlexOlmo as the basis for FlexMoRE, a more efficient modular LLM architecture that lets institutions contribute specialized experts trained on sensitive or proprietary data without sharing that data—and run the resulting models on highly accessible hardware.