Scaling up BERT-like model Inference on modern CPU - Part 2

Hugging Face Blog · 4 years ago

Intel's Ice Lake Xeon CPUs achieve up to 75% faster inference on NLP tasks compared to the previous Cascade Lake generation through hardware improvements like new instructions and PCIe 4.0 support. The performance gains come from software optimizations including Intel's oneAPI libraries (oneMKL, oneDNN, oneTBB) and framework-specific tuning of memory allocation, parallelization, and mathematical operators. Users can extract these improvements by enabling oneDNN in TensorFlow via an environment variable or through PyTorch's native integration, along with tuning parallelization settings like OpenMP configuration.