Multimodal neurons in artificial neural networks

OpenAI's CLIP model was trained on a simple task: match images to captions scraped from the internet. But poke around inside its guts and something stranger shows up. Researchers found individual neurons that light up for a single concept no matter how it's represented — a photo of a spider, a cartoon spider, the word 'spider' scrawled in someone's handwriting. One neuron, they claim, does not care about pixels versus letters versus symbols. It just cares about spiders.

This isn't a totally novel discovery in itself. Neuroscientists have documented similar behavior in humans for years, most famously the so-called Halle Berry neuron, a single cell in one patient's brain that fired for photos of the actress, sketches of her, and even the text of her name. What's new is finding the same pattern emerge, unprompted, inside a neural network trained purely on image-caption pairs pulled from the web, with no instruction to organize information this way.

The practical upside is that this multimodal wiring seems to explain why CLIP is unusually good at handling weird visual tricks. Feed it a photo of a poodle with the word 'iPod' pasted across it, and CLIP will often classify it as an iPod, not a dog — because the text-recognizing neurons override the object-recognizing ones. That's a party trick, but it also reveals how the model actually reasons, treating text overlays as evidence just as strong as visual shape.

The less charming part is what else these neurons pick up. OpenAI's team found neurons associated with regions, religions, and emotions that carry the same messy stereotypes baked into the training data — a 'Middle East' neuron tangled up with terrorism imagery, for instance, or neurons that blend immigration with Latin America. Because the concepts are abstracted and shared across modalities, a biased association learned from one type of input can bleed into how the model treats a completely different one, like a caption or a hand-drawn sketch.

OpenAI frames this as an early but important window into interpretability: if you can find the individual neurons carrying a concept, you can in theory audit what a model has actually learned rather than just testing its outputs. That's a genuinely useful research direction. It also means the industry's biggest models are quietly encoding human prejudice at a structural level, not just a superficial one, and no amount of dataset curation after the fact fixes wiring that forms during training itself.