A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using transformers, accelerate and bitsandbytes

Hugging Face Blog · 3 years ago

Hugging Face integrated LLM.int8(), a quantization technique that compresses large language models to 8-bit precision while maintaining inference quality. The method reduces memory requirements by 4x compared to standard 32-bit storage, allowing BLOOM-176B to run on fewer GPUs. This enables inference of 176-billion-parameter models on accessible hardware without performance degradation.