Multimodal neurons in artificial neural networks
OpenAI Blog
Researchers identified neurons in the CLIP multimodal model that activate in response to the same concept regardless of whether it appears as a literal image, a symbol, or a conceptual representation. These neurons demonstrate consistent response patterns across different modalities, suggesting a unified internal representation of concepts. This finding helps explain why CLIP performs well on unusual visual interpretations and provides a method to audit what associations and biases multimodal models encode.
Why it matters
We’ve discovered neurons in CLIP that respond to the same concept whether presented literally, symbolically, or conceptually. This may explain CLIP’s accuracy in classifying surprising visual renditions of concepts, and is also an important step toward understanding the associations and biases that CLIP and similar models learn.
