Generalized Visual Language Models

Lilian Weng · 4 years ago

Researchers have developed multiple approaches to extend pre-trained language models to process visual information alongside text, grouping these vision-language models (VLMs) into four categories: joint training with image-text embeddings, frozen language model prefixes, cross-attention fusion mechanisms, and combined models without training. Notable models include VisualBERT trained on MS COCO with masked language modeling and sentence-image prediction objectives, SimVLM mixing 4,096 image-text pairs with 512 text-only documents per batch, and CM3 trained on close to 1 trillion tokens of web data tokenized to 256 tokens per image. These approaches enable language models to perform vision-language tasks like image captioning and visual question-answering while preserving or leveraging existing pre-trained linguistic capabilities.