Accelerating PyTorch distributed fine-tuning with Intel technologies

Hugging Face · 4 years ago · 35

Intel has published guidance on accelerating PyTorch distributed fine-tuning using Intel Xeon Scalable CPU servers with Ice Lake architecture and optimized software libraries. The baseline single-node BERT training job on the MRPC dataset takes 7 minutes and 46 seconds before distributed optimization. The approach involves using Intel's extension for PyTorch and the oneAPI Collective Communications Library to reduce network bottlenecks when distributing training across multiple CPU-based server clusters.