Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval
Hugging Face 2 years ago 25
Researchers introduced binary and scalar quantization methods that convert high-precision embeddings into lower-precision formats, reducing memory and storage requirements without proportional performance loss. Binary quantization reduces embeddings from float32 to 1-bit values, achieving 32x memory reduction while preserving approximately 96% retrieval performance when combined with a rescoring step, and the Hamming Distance comparison between binary embeddings requires only 2 CPU cycles. Organizations storing 250 million embeddings can reduce monthly infrastructure costs from thousands of dollars to a fraction of that amount and dramatically accelerate retrieval speed through these quantization approaches.