Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia

Hugging Face Blog · 4 years ago

Hugging Face Transformers and AWS released a tutorial for deploying BERT models on AWS Inferentia chips, custom hardware designed to accelerate inference workloads. AWS Inferentia delivers up to 80% lower cost per inference and 2.3X higher throughput than comparable GPU instances, with the tutorial achieving 5-6ms latency for sequence length 128. Companies moving BERT to production can reduce inference costs and increase throughput by using Inferentia instead of GPUs or CPUs.