Multimodal Embedding & Reranker Models with Sentence Transformers

Hugging Face's Sentence Transformers library has quietly been the backbone of a huge chunk of semantic search and RAG tooling for years, mostly for text. With version 5.4, that changes. The library now handles multimodal embedding and reranking models through the exact same encode() and rank() calls developers already know, meaning you can throw images, audio clips, and video at it alongside plain strings without learning a new interface.

The core idea is a shared embedding space. Models like Qwen3-VL-Embedding-2B map text and images into the same vector space, so a query like "a green car parked in front of a yellow building" scores 0.51 against a matching car photo and only 0.11 against a picture of a bee. That's the useful part. The less flattering part is what the post calls the modality gap: cross-modal similarity scores rarely climb near 1.0 even for perfect matches, because embeddings from different modalities cluster in their own neighborhoods of the space. Ordering still holds up for retrieval purposes, but absolute scores need to be read with that caveat in mind.

On the reranking side, CrossEncoder models like Qwen3-VL-Reranker-2B and jinaai/jina-reranker-m0 score relevance between pairs where either side can be text, image, or a text-image combo. In the demo, the reranker correctly ranked an actual car photo above a text-plus-image description of a car in a European city, and dumped a bee image to the bottom of the list. Rerankers are pitched as more accurate but slower than embedding models, which is why the library leans on a two-stage pattern: cheap embedding search over a large corpus first, then a reranker cleans up the top handful of results.

Practically, none of this is free. Qwen3-VL-2B needs roughly 8GB of VRAM, and the 8B variants want closer to 20GB, so CPU users are steered toward CLIP-style or text-only models instead. Installation is now modular too, with separate pip extras for image, audio, and video support so you're not dragging in dependencies you don't need. The library also auto-detects which modalities a loaded model actually supports, via a simple modalities property, which matters since not every multimodal model handles every input type.

What's notable here isn't a new model — it's that Hugging Face made mixed-modality retrieval a drop-in upgrade rather than a rewrite. Encode a URL, a file path, or a dict combining text and image, and it all funnels into the same internal message format. For teams already running Sentence Transformers pipelines, that's a much lower bar to start building things like visual document search or multimodal RAG than building a bespoke stack from scratch.