Accelerate StarCoder with 🤗 Optimum Intel on Xeon: Q8/Q4 and Speculative Decoding

Hugging Face · 2 years ago · 41

Hugging Face and Intel optimized the StarCoder code generation model for Intel Xeon processors using quantization and speculative decoding techniques. The optimized model achieved 7.30x speedup in token generation latency by combining 8-bit quantization on both a large target model and a 164-million-parameter draft model with assisted generation. Users can now run the optimized StarCoder on Xeon CPUs with minimal accuracy loss by replacing standard model classes with IPEXModelForCausalLM from the optimum-intel library.