Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese
Qwen
Researchers developed Chinese CLIP, a vision-language model adapted for Chinese by using a two-stage pretraining method with frozen and unfrozen encoders on 200 million image-text pairs from public datasets. The model achieved the best performance on Chinese cross-modal retrieval benchmarks like MUGE, with notably larger improvements over previous models on native Chinese datasets than on English-native datasets. This language-specific approach addresses limitations of directly translating English CLIP outputs and enables better understanding of Chinese cultural concepts in vision-language applications.
Why it matters
CLIP1 is a phenomenal playmaker in vision and multimodal representation learning. It plays not only as a foundation model but also a bridge between vision and language. It has triggered a series of research in different fields, especially text-to-image generation. However, we find that there is a necessity for a language-specific CLIP for applications, especially cross-modal retrieval, and there is no opensourced Chinese CLIP with good performance. We therefore launched this project to promote the Chinese multimodal representation learning.
