TLDRocket
Sign in

Hardware & Infrastructure

261 summarised stories in Hardware & Infrastructure, each linking back to the original source. Browse all topics →

Wednesday, 16 March 2022

Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia

Hugging Face 4 years ago 30

Hugging Face Transformers and AWS released a tutorial for deploying BERT models on AWS Inferentia chips, custom hardware designed to accelerate inference workloads. AWS Inferentia delivers up to 80% lower cost per inference and 2.3X higher throughput than comparable GPU instances, with the tutorial achieving 5-6ms latency for sequence length 128. Companies moving BERT to production can reduce inference costs and increase throughput by using Inferentia instead of GPUs or CPUs.

The daily briefing

Every AI story that matters, in your inbox by 8am.

TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.