NeoMME: an efficient Multimodal-native and Multilingual Encoder

Hugging Face’s NeoMME is trying to do something a little unfashionable: make multimodal models smaller, simpler, and still useful. The new family comes in 260M and 800M parameter versions, and both are built around a single bidirectional Transformer that handles text tokens and image patches together. No separate pretrained vision tower. No causal language model bolted on top.

That design choice matters because the target here isn’t chatty image generation. It’s retrieval, classification, and token labeling — jobs that don’t need an autoregressive decoder chewing through tokens one by one. NeoMME is trained from scratch with a masked discrete-diffusion objective, using multilingual text, code, mathematics, natural images, and document images. Each model sees about 524 billion packed input tokens, with 290 billion from text-only examples.

The architecture is built for documents rather than generic pretty pictures. Images keep their aspect ratio, get split into 32×32 patches, and share the same path as text. The models have a 16,384-token context length, with sliding-window attention in most layers and global attention in every sixth layer plus the final layer. Hugging Face also trained a 131k-token BPE tokenizer from scratch on multilingual text, code, mathematics, and machine-produced image transcripts.

The retrieval story is the sharper part. NeoMME-Retriever adds two heads on top of the backbone: a dense head and a late-interaction head, both returned in one forward pass. That lets the same model serve fast ANN search or finer reranking without a second encoding step. On ViDoRe v3, the 260M model scores 0.523 nDCG@10, the best among evaluated models under 800M parameters, while the 800M version reaches 0.556.

Speed and storage are where the engineering gets very practical. At 2048×2048 input size on an NVIDIA L40S, the 260M model encodes about 51 pages per second, roughly twice ColModernVBERT’s 26 pages per second. And with hierarchical token pooling plus asymmetric quantization, late-interaction storage can drop from about 1.5 MB per page to 6 kB, a 255× reduction, while keeping more than 95% of baseline nDCG@10. The checkpoints are out under Apache 2.0, and the models are already in Hugging Face Transformers.