Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

Hugging Face Blog · 1 year ago

Abu Dhabi's Technology Innovation Institute released Falcon-H1, a series of six open-source language models combining Transformer attention with State Space Model architecture to improve efficiency and long-context performance. The smallest model, Falcon-H1-0.5B, delivers performance comparable to typical 7B models from 2024, while the 34B variant matches or exceeds much larger competitors like Qwen2.5-72B on long-context tasks. The hybrid design enables faster inference and lower memory usage across deployments ranging from edge devices to large-scale systems, with support for 18 languages and 256K context length.