Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

arXiv cs.AI · 18 hours ago

Researchers studied how long to train expert models before merging them into a single model, testing five merging methods across domains like Math and Code on models ranging from 0.8B to 4B parameters. Sparsification-based merging methods performed best when experts were trained 200-500% past their optimal validation loss, while simple averaging degraded with such overtraining. The findings suggest that optimal training duration depends on the merging method chosen, rather than stopping at standard validation loss as current practice dictates.