Mastering Long Contexts in LLMs with KVPress
Hugging Face
NVIDIA released KVPress, a toolkit that shrinks the memory-hungry KV cache LLMs use for long contexts. That 1M-token cache can eat 330GB of memory; this cuts it down without gutting accuracy.
Based on reporting by Hugging Face — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Long context windows sound great until you check the memory bill. Feed Llama 3-70B a million tokens in bfloat16 and the KV cache alone wants roughly 330GB, on top of the 140GB the model weights already take. Add it up and you're near 470GB just to keep a single long conversation alive, with the cache eating about 70% of that total. That math is why context windows have grown a lot faster than anyone's ability to actually run them cheaply.
NVIDIA's answer is KVPress, a Python toolkit built to compress that cache on the fly rather than let it balloon linearly with sequence length. The library ships with what it calls presses, algorithms that score each key-value pair in every attention head and prune the least useful ones. KnormPress, for instance, drops pairs with the smallest key norm. SnapKVPress instead looks at which pairs got the least attention from recent queries and cuts those. Both hook directly into the model's attention layers through forward hooks, so the compression happens during generation without you rewriting the model.
The timing matters here. KVPress targets the pre-filling phase, the moment right when the cache is at its biggest, which is exactly when squeezing it pays off most. On Llama 3.1 8B, applying a 50% compression ratio at 128k context length dropped peak GPU memory from 45GB down to 37GB, and decoding speed climbed from 11 to 17 tokens per second on an A100. That's not a rounding error — it's the difference between a setup that fits on your hardware and one that doesn't.
The toolkit also plugs neatly into an existing transformers pipeline, so swapping in a press like ExpectedAttentionPress takes about four lines of code, no exotic dependencies required. NVIDIA has already benchmarked more than a dozen presses against datasets like RULER, InfiniteBench, and Loogle, and the standout so far combines AdaKVPress with ExpectedAttentionPress, an unpublished technique from the KVPress team itself. Compression isn't free, though — push the ratio too high and accuracy starts slipping, which is exactly the trade-off the benchmarks are meant to expose rather than hide.
KVPress won't be the last word on this problem, but it gives researchers a shared, modular place to test new pruning ideas instead of everyone reinventing cache compression from scratch. For anyone actually trying to run million-token contexts without renting a small data center, that's the kind of unglamorous infrastructure work that ends up mattering more than the next benchmark headline.
My take — AI-written commentary, not fact-checked reporting
This is the boring-but-essential kind of release that keeps the whole long-context hype train from derailing on hardware costs, and I like that NVIDIA open-sourced it instead of burying it in a proprietary API. The real story isn't the compression ratios, it's that memory, not model quality, is now the actual bottleneck for long-context LLMs — and tools like this are quietly becoming as important as the models themselves.
Read more about this at: Hugging Face