Mastering Long Contexts in LLMs with KVPress
Hugging Face Blog 1 year ago
NVIDIA released KVPress, a toolkit that compresses the key-value cache used during language model inference to reduce memory consumption for long-context processing. Processing 1 million tokens with Llama 3-70B requires 327.6GB for the KV cache alone, but KVPress with a 50% compression ratio reduced peak memory usage from 45GB to 37GB on a 128k token context length. The compression enables faster decoding speeds, improving from 11 to 17 tokens per second on an A100 GPU, making long-context inference more practical for resource-constrained deployments.