Visual Salamandra: Pushing the Boundaries of Multimodal Understanding

Hugging Face Blog · 1 year ago

The Language Technologies Lab released Visual Salamandra, a multimodal extension of its Salamandra large language model that processes images, video, and text inputs. The model uses a 7 billion parameter architecture combining Google's SigLIP encoder with a custom MLP projector trained across four phases on 6.1 million instruction-tuning instances, including 842,000 text-only samples. Visual Salamandra supports tasks including visual question answering, optical character recognition, document analysis, and mathematical reasoning, with emphasis on European language representation.