Case Study: Millisecond Latency using Hugging Face Infinity and modern CPUs

Hugging Face · 4 years ago · 5

Hugging Face published performance benchmarks for Infinity, a now-discontinued containerized inference solution that optimized Transformer models for CPU deployment. Running a DistilBERT model on Intel Ice Lake processors with 2 physical cores achieved 248 requests per second at batch size 1 with sequence length 8, compared to 49 requests per second with vanilla Transformers. The optimization enabled latency as low as 1-4 milliseconds for sequences up to 64 tokens, though Infinity itself has been replaced by Inference Endpoints and open-source optimization libraries.