Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention
Ahead of AI Sebastian Raschka, PhD ● Covered by 2 sources
Recent open-weight LLM releases including Gemma 4, DeepSeek V4, and others have adopted architectural techniques like KV sharing across layers, per-layer embeddings, and compressed attention to reduce memory and compute costs for long-context processing. Gemma 4 E2B achieves approximately 2.7 GB of KV cache savings at 128K context length through cross-layer KV sharing that allows later transformer layers to reuse key-value tensors from earlier layers. These efficiency-focused design changes enable smaller models to handle longer contexts and reduce memory requirements, which becomes critical as reasoning models and agent workflows maintain more tokens during inference.
Why it matters
From Gemma 4 to DeepSeek V4, How New Open-Weight LLMs Are Reducing Long-Context Costs
