Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

Hugging Face Blog · 3 months ago

The Sentence Transformers library now supports training and finetuning multimodal embedding models that handle text, images, audio, and video across tasks like visual document retrieval and semantic search. A finetuned Qwen3-VL-Embedding-2B model achieved an NDCG@10 score of 0.947 on visual document retrieval, compared to the base model's 0.888 and outperforming all tested competitors including models four times larger. Users can now train domain-specific multimodal models using the same SentenceTransformerTrainer component used for text-only models, with automatic handling of image preprocessing and support for multiple loss functions including CachedMultipleNegativesRankingLoss.