TurboQuant: Redefining AI efficiency with extreme compression
Google Research ● Covered by 2 sources
Google Research built TurboQuant, a way to shrink AI memory data to 3 bits with zero accuracy loss. That's the kind of compression that usually costs you accuracy — this doesn't.
Based on reporting by Google Research — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Every big language model carries a hidden weight problem. It's not the parameters everyone talks about, it's the key-value cache, the scratchpad that stores context so a model doesn't have to reread everything it's already seen. That cache balloons fast with long context windows, and squeezing it down with standard vector quantization usually taxes you extra bits per number just to store the compression math itself. Google Research says it found a way around that tax, and they're calling it TurboQuant.
The trick is really two tricks stacked together. First, PolarQuant rotates data vectors and reframes them in polar coordinates instead of the usual X-Y-Z grid, turning a messy, constantly-shifting normalization problem into something that fits a fixed, predictable pattern, sort of like knowing every taxi ride will start from the same corner. That alone kills most of the memory overhead. Second, a companion method called QJL mops up the small residual error using the Johnson-Lindenstrauss transform, boiling leftover noise down to a single sign bit — plus or minus — with no additional storage cost at all.
The numbers are the part that make this worth a second look. Tested on Gemma and Mistral across long-context benchmarks like LongBench, RULER, and Needle In A Haystack, TurboQuant compressed the KV cache by at least 6x with what the researchers describe as zero accuracy loss on the needle-in-haystack tasks. Push it to 3 bits and there's still no training or fine-tuning required. On H100 GPUs, the 4-bit version computed attention logits up to 8 times faster than uncompressed 32-bit keys, which is the kind of speedup that shows up directly in inference cost, not just a benchmark chart.
It's not only a cache trick, either. The same math applies to vector search — the technology behind semantic search, recommendation engines, and anything that needs to find the closest match among billions of embeddings. Google says TurboQuant beat established methods like PQ and RabbiQ on recall, even though those rivals lean on large codebooks and dataset-specific tuning that TurboQuant doesn't need. Google frames this as groundwork for keeping products like Gemini and its search systems fast as they lean harder into vector-based retrieval, and the paper backs the claims with theoretical proofs showing the methods sit close to known lower bounds, not just favorable cherry-picked test runs.
My take — AI-written commentary, not fact-checked reporting
I'm always suspicious when a paper claims a compression trick with 'zero accuracy loss' — that phrase usually hides an asterisk somewhere in the fine print. But the theoretical backing here, not just benchmark screenshots, is what makes me take it seriously; provably-efficient methods age a lot better than ones tuned to look good on LongBench. If this holds up outside Google's own stack, it's a bigger deal for the industry's power bill than most of the flashy model releases this quarter.
Read more about this at: Google Research