Multimodality and Large Multimodal Models (LMMs)

Chip Huyen · 2 years ago · 48

Large Multimodal Models (LMMs) combine language models with the ability to process multiple data types like text, images, and audio, with CLIP and Flamingo serving as foundational examples demonstrating how to align different modalities into shared embedding spaces. CLIP achieved competitive zero-shot performance on image classification tasks and has been adopted as the image encoder in systems like Flamingo and LLaVA. The shift toward multimodal systems enables practical applications across healthcare, robotics, e-commerce, and accessibility use cases that require processing mixed data types.