Quanto: a PyTorch quantization backend for Optimum

Hugging Face · 2 years ago · 28

Hugging Face released Quanto, a PyTorch quantization backend for Optimum that reduces model size and computational costs by converting weights and activations to lower-precision data types like int8 or float8. The tool supports int2, int4, int8, and float8 weights across any model architecture and device (CPU, GPU, Apple Silicon), with accelerated int8-int8 and mixed-precision matrix multiplications on CUDA hardware. Quanto integrates directly into the transformers library, allowing developers to quantize models in a few lines of code without restricting themselves to specific model configurations or device types.