Mastering Long Contexts in LLMs with KVPress

Hugging Face · 1 year ago · 40

NVIDIA released KVPress, a toolkit that compresses the key-value cache used during language model inference to reduce memory consumption for long-context processing. Processing 1 million tokens with Llama 3-70B requires 327.6GB for the KV cache alone, but KVPress with a 50% compression ratio reduced peak memory usage from 45GB to 37GB on a 128k token context length. The compression enables faster decoding speeds, improving from 11 to 17 tokens per second on an A100 GPU, making long-context inference more practical for resource-constrained deployments.