Scaling up BERT-like model Inference on modern CPU - Part 2

Hugging Face · 4 years ago · 42

Intel's Ice Lake Xeon CPUs achieve up to 75% faster inference on NLP tasks compared to the previous Cascade Lake generation through hardware improvements like new instructions and PCIe 4.0 support. The performance gains come from software optimizations including Intel's oneAPI libraries (oneMKL, oneDNN, oneTBB) and framework-specific tuning of memory allocation, parallelization, and mathematical operators. Users can extract these improvements by enabling oneDNN in TensorFlow via an environment variable or through PyTorch's native integration, along with tuning parallelization settings like OpenMP configuration.