Blazing Fast SetFit Inference with 🤗 Optimum Intel on Xeon

Hugging Face Blog · 2 years ago

SetFit, a framework for few-shot fine-tuning of sentence transformer models, can now run 7.8x faster on Intel Xeon CPUs using Optimum Intel's quantization optimization. The optimization reduces model latency from 15.69ms to 4.55ms at batch size 1 while shrinking model size from 127.32MB to 44.65MB with minimal accuracy loss (88.4% to 88.1%). This enables production-grade deployment of SetFit solutions on Intel hardware without requiring expensive GPU infrastructure.