Read the History, Not Just the Code

When someone joins a codebase, the first instinct is to read the code and trust what’s there. Then you hit the weird conditional, the useless parameter, the comment that says “do not remove,” and the real explanation is nowhere in sight. The code shows the result. The history explains the mess.

That’s the lens Naomi Saphra is applying to language models. In a Quanta interview, the Harvard Kempner Institute researcher argued that interpretability is looking in the wrong place if it only inspects a trained model after the fact. Her target is the training process itself, especially stochastic gradient descent, the repeated nudging that shapes the model over time.

She frames it like evolutionary biology, not neuroscience. The point is simple enough to sting: if you do not understand where a model came from, you do not really understand why it works. That sounds familiar to anyone who has ever tried to make sense of a codebase without its commit log. The present state is not enough. You want to know which files moved together, which workaround arrived when, and what was broken at the time.

Her best example is almost embarrassing in its familiarity. In image classifiers, some neurons look like they are doing something important because they fire for a single category, such as cats. Then researchers suppress those neurons during training, and the model improves. The thing that looked central was actually vestigial — a leftover from earlier training, more burden than brain.

She makes a similar point about early decisions sticking around. If a model is trained first on clean structured text and then on messier informal text, it can cling to the easy rule and start memorising exceptions instead of learning the general one. That is technical debt with a lab coat on. Early choices shape what the system can become, and later fixes do not always undo them.

Saphra also points to two practical problems. Training runs vary, so the same setup can produce different models, and she treats that variation as evidence, not just noise. And companies often do not provide intermediate checkpoints, which leaves researchers staring at the final state and trying to debug a failure from the crash alone. That is a very modern kind of blindness, and it is not limited to AI.