Build specialized multimodal models with LFM3

Hugging Face is positioning LFM3 as the starting point for specialized AI, not a giant one-size-fits-all model. The new family folds text, vision, audio, omni, and decision models onto a shared backbone, all in the 2B-parameter range. The selling point is flexibility: pick the modality you need, tune it on your own data, and deploy it where the app runs.

The main architectural shift is inside LFM3-1.7B-Base. Hugging Face says a new tri-hybrid design lets LFM3-1.7B outperform LFM2.5-2.6B while using 35% fewer parameters. That backbone mixes 13 gated short-convolution layers, 5 sliding-window attention layers, and 6 global attention layers across 24 layers total. It also uses head-wise gated attention, partial RoPE in the global layers, and sandwich norm to keep activations under control.

The edge story is a big part of the release. The text model fits in about 1 GB at 4-bit precision, and its 256K context window is meant to support on-device agents. Hugging Face also says the model is faster over long contexts: at 8K tokens, LFM3-1.7B is about 25% faster than LFM2-1.2B on edge devices. The backbone was pre-trained on 22T tokens, then went through a final 3.5T-token mid-training stage with more agentic traces and tool calls.

From that shared base come six models. LFM3-VL-2B adds video input for vision tasks. LFM3-Audio-2B handles voice chat, speech recognition, speech synthesis, and translation across 18 languages, with time to first token as low as 50 ms. LFM3-Omni-2B combines text, vision, and audio in one model for speech output and video understanding. The d1 models take a different route entirely, returning calibrated decisions instead of generated text, with d1-2B focused on text, pictures, and video and d1-omni-2B adding audio.

Hugging Face is also leaning hard on efficiency claims for the new encoders and codec. The vision encoder uses 3x less compute than the SigLIP2 encoder in earlier LFM VL models, and the audio stack can stream speech with LiquidCodec at about 87x real time on a single CPU thread on Apple M4 Max. The message underneath all of it is pretty clear: if a model can be smaller, faster, and easier to specialize, Hugging Face would rather ship that than another bloated generalist.