Accelerated Inference with Optimum and Transformers Pipelines

Hugging Face · 4 years ago · 33

Hugging Face released Optimum 1.2 with inference support for Transformers pipelines using ONNX Runtime, allowing users to accelerate model inference with a compatible API. In a tutorial example converting RoBERTa for question-answering, quantization reduced model size from 473 MB to 291 MB while maintaining prediction accuracy. Users can now replace standard Transformers model classes with optimized ORTModel equivalents to run faster inference on production workloads.