KV Cache from scratch in nanoVLM

Hugging Face · 1 year ago · 12

The nanoVLM team implemented KV caching, a technique that caches key and value matrices during transformer inference, in their small Vision Language Model codebase. The optimization achieved a 38% speedup in generation by eliminating redundant recomputation of attention keys and values for previously processed tokens. The implementation separates generation into a prefill phase that processes the full prompt once and a decode phase that generates tokens incrementally using the cached values.