Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers
Hugging Face Blog
The Sentence Transformers library now supports training and finetuning multimodal embedding models that handle text, images, audio, and video across tasks like visual document retrieval and semantic search. A finetuned Qwen3-VL-Embedding-2B model achieved an NDCG@10 score of 0.947 on visual document retrieval, compared to the base model's 0.888 and outperforming all tested competitors including models four times larger. Users can now train domain-specific multimodal models using the same SentenceTransformerTrainer component used for text-only models, with automatic handling of image preprocessing and support for multiple loss functions including CachedMultipleNegativesRankingLoss.