Sentence Transformers 5.4 Brings Multimodal Embeddings to RAG

The latest Sentence Transformers release adds native support for multimodal embeddings, allowing text, images, audio, and video to be encoded and compared in a shared embedding space. This enables new use cases for Retrieval Augmented Generation (RAG) systems.

Details

The article discusses how the latest Sentence Transformers release, version 5.4, introduces a fundamental change by adding native support for multimodal embeddings. This means the same encoding and similarity computation workflows can now handle text, images, audio, and video inputs, mapping them into a shared embedding space. This addresses the limitations of traditional text-only embedding models, which struggle with queries involving visual content. With multimodal embeddings, RAG systems can now retrieve relevant images, screenshots, diagrams, and other visual documents alongside text, without the need for separate image search pipelines or OCR preprocessing. The article also highlights the practical impact of this change, including use cases like visual document RAG, cross-modal search, and multimodal deduplication. However, it notes that production-ready multimodal RAG systems still require further advancements in areas like index efficiency, chunking strategies for non-text media, and multimodal evaluation benchmarks.