TurboQuant: Redefining AI efficiency with extreme compression

Google Research · 3 months ago

Researchers introduced TurboQuant, a vector compression algorithm designed to reduce the memory footprint of key-value caches in AI models while maintaining accuracy. The method achieves 6x reduction in key-value memory size and up to 8x speedup in attention computation on H100 GPUs when compressing to 3-4 bits without requiring model retraining. The technique enables faster vector search and more efficient large-scale AI applications by eliminating memory overhead inherent in traditional quantization methods.