Modular LLMs at scale: how FlexOlmo is helping to pool national expertise without pooling sensitive data

Allen Institute (AI2) · 1 month ago · 6

Researchers at the University of Southern Denmark built FlexMoRE, a modular language model that allows separate institutions to train specialized components independently without sharing sensitive data. FlexMoRE reduces memory demands to less than one-third of the original FlexOlmo model while maintaining performance by using smaller low-rank adapters instead of full-size experts. This enables distributed model training for lower-resource languages like Danish while preserving privacy, allowing organizations with restricted data to contribute expertise to shared models.