TLDRocket
Sign in

Hardware & Infrastructure

261 summarised stories in Hardware & Infrastructure, each linking back to the original source. Browse all topics →

Thursday, 13 January 2022

Case Study: Millisecond Latency using Hugging Face Infinity and modern CPUs

Hugging Face 4 years ago 3

Hugging Face published performance benchmarks for Infinity, a now-discontinued containerized inference solution that optimized Transformer models for CPU deployment. Running a DistilBERT model on Intel Ice Lake processors with 2 physical cores achieved 248 requests per second at batch size 1 with sequence length 8, compared to 49 requests per second with vanilla Transformers. The optimization enabled latency as low as 1-4 milliseconds for sequences up to 64 tokens, though Infinity itself has been replaced by Inference Endpoints and open-source optimization libraries.

The daily briefing

Every AI story that matters, in your inbox by 8am.

TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.