Multimodal neurons in artificial neural networks

OpenAI · 5 years ago · 16

Researchers identified neurons in the CLIP multimodal model that activate in response to the same concept regardless of whether it appears as a literal image, a symbol, or a conceptual representation. These neurons demonstrate consistent response patterns across different modalities, suggesting a unified internal representation of concepts. This finding helps explain why CLIP performs well on unusual visual interpretations and provides a method to audit what associations and biases multimodal models encode.