“We were looking at it wrong”: The explosive claim that’s upended the AI world

Nick Cammarata, a researcher at AI startup Goodfire, has set off a fresh argument over whether large language models are actually understandable. His long blog post argues that neural networks may be far less opaque than the field has assumed, and that we may be able to monitor and control them better than expected. That idea hit a nerve because the AI industry has spent the summer watching agents from top firms, including OpenAI and Anthropic, break containment and create public trouble without their makers always knowing what they were doing.

Cammarata is not some outsider tossing grenades from the sidelines. He previously worked at OpenAI and helped develop techniques for peering inside models, including methods that inspect individual neurons. That work helped shape mechanistic interpretability, the research program built around understanding what goes on inside these systems. For years, the prevailing assumption has been that neural networks are tangled, messy and maybe impossible to fully unwind. Researchers found early evidence for that view in models like Inception v1, where one neuron appeared to respond to both cars and cats, a pattern that fed the idea of superposition: unrelated concepts crammed together inside the same neuron.

Cammarata says he used to believe that too. But after spending long days staring at neurons at Goodfire — he says he spent about 10 hours a day, seven days a week, for most of the last year — he now thinks the picture is less hopeless. His argument is not that every model is suddenly transparent. It is that there is structure there, and that if researchers keep looking carefully enough, they can map it. He says he was able to map out whole circuitry, and that what once looked like a dead end may have been a case of researchers expecting confusion and then finding it.

His Inception v1 example is the cleanest version of that claim. Instead of asking why cars and cats were bundled together, he looked for what they had in common and decided the answer was shape: a V. The neuron also lit up for things like the corner of a pool table, the pointed nose of a Japanese high-speed train, and a fire boat shooting water upward, while round O shapes tended to switch it off. On that reading, the model was not storing random concepts in one place. It was storing a pattern. Cammarata says that discovery made him rethink the whole enterprise. “We were looking at it wrong,” he says.

Not everyone is buying the pivot. Stanford’s Chris Potts says the post does not make the underlying problems easier, even if people on social media have treated it like a sweeping breakthrough. Jacob Hilton at the Alignment Research Center says researchers still understand less than 0.01% of top-tier LLMs, and Neel Nanda at Google DeepMind says neuron-level insights are interesting but not the same as understanding the whole system. Cammarata’s team plans to publish technical results on modern LLMs soon. For now, the big claim is simple enough: maybe this is not a nonstarter after all.