Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention

Ahead of AI · 2 months ago

Recent open-weight LLM releases including Gemma 4, DeepSeek V4, and others have adopted architectural techniques like KV sharing across layers, per-layer embeddings, and compressed attention to reduce memory and compute costs for long-context processing. Gemma 4 E2B achieves approximately 2.7 GB of KV cache savings at 128K context length through cross-layer KV sharing that allows later transformer layers to reuse key-value tensors from earlier layers. These efficiency-focused design changes enable smaller models to handle longer contexts and reduce memory requirements, which becomes critical as reasoning models and agent workflows maintain more tokens during inference.