[AINews] Agents disprove Polysemanticity; greatly improve Interpretability & Alignment
Latent Space
Goodfire says agents can make messy neural nets readable. If true, a lot of old “superposition” doom talk just got a lot smaller.
Based on reporting by Latent Space — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
The most interesting thing in this roundup isn’t a new model at all. It’s Nick Cammarata’s new Goodfire essay, which argues that many of the model internals people filed under “polysemantic” or “superposed” may not be as tangled as they looked. His line is simple enough: keep probing with more agentic analysis, and a lot of those knots start to come apart.
That’s a sharp reversal from the old story. In 2023, Goodfire’s automated-interpretability work at OpenAI reportedly found only about 2% of neurons legible. Cammarata now says many famous examples were misread, and that models all the way up to trillions of parameters are more readable than the field assumed.
The reaction split fast. Tszzl and others treated it like a real moment. Neel Nanda didn’t move off his core view; he still expects representational superposition, and he still thinks weak activations can hold memorized facts. Leo Gao pushed the focus toward neuron simulation measured with KL. Jake Mendel pointed to the neglected tail of non-max activations. Pfau added the obvious caution: the paper itself still isn’t out.
So the claim here isn’t that interpretability is solved. It’s that the cheap fatalism around it may be dying. If more agentic analysis can keep stripping away supposed examples of polysemanticity, then the field gets a better shot at alignment work that isn’t guessing in the dark. That’s a much more interesting outcome than another round of hand-wringing about opaque weights.
My take — AI-written commentary, not fact-checked reporting
The cleanest takeaway is that a lot of interpretability people may have been overrating how mysterious these models are. That’s embarrassing, but useful. The AI safety world has spent years sounding like the locks are made of black magic; it turns out some of them are just bad lighting and a flashlight still works.
Read more about this at: Latent Space
