Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel

Hugging Face · 3 months ago · 9

NVIDIA released NeMo AutoModel, an open library that optimizes fine-tuning of Mixture-of-Experts models by building on HuggingFace Transformers v5 with additional techniques like Expert Parallelism and DeepEP fused dispatch. NeMo AutoModel achieves 3.4-3.7x higher training throughput and 29-32% less GPU memory consumption compared to native Transformers v5 while maintaining API compatibility through a single import change. Users can now fine-tune large MoE models like Nemotron 3 Ultra 550B across multiple nodes or smaller models on single nodes with the same from_pretrained() API without code modifications.