Quanto: a PyTorch quantization backend for Optimum

Hugging Face Blog · 2 years ago

Hugging Face released Quanto, a PyTorch quantization backend for Optimum that reduces model size and computational costs by converting weights and activations to lower-precision data types like int8 or float8. The tool supports int2, int4, int8, and float8 weights across any model architecture and device (CPU, GPU, Apple Silicon), with accelerated int8-int8 and mixed-precision matrix multiplications on CUDA hardware. Quanto integrates directly into the transformers library, allowing developers to quantize models in a few lines of code without restricting themselves to specific model configurations or device types.