How Goodfire used Ai2’s open post-training stack to trace unwanted model behavior

Preference training is supposed to make large language models more helpful, safer, or more compliant. In practice, it can also nudge a model in directions nobody planned. A tweak that improves one area can quietly weaken a safeguard somewhere else, and that makes post-training feel less like tuning and more like forensic work.

Goodfire, an interpretability research company, used Ai2’s fully open post-training stack to show a different way through that mess. The team predicted behavioral changes before a full training run, traced a safety regression back to individual preference examples, and tried targeted fixes that aimed to reduce the problem without giving up broader capability gains.

That was only possible because Ai2 exposes the whole pipeline. Dolci, Ai2’s open preference dataset, includes the preferred and rejected responses used to train Olmo 3. Olmo also ships with intermediate checkpoints and reproducible training recipes, while OLMES provides a standardized evaluation suite for tracking capability changes across a consistent set of benchmarks. Put together, those pieces let Goodfire work on the training process itself instead of staring at the finished model and guessing.

Leon Bergen, a Goodfire researcher and UC San Diego associate professor who co-authored the paper, said the reproducible Olmo pipeline meant the team could test a component and compare the result against Ai2’s official checkpoints. That matters because the question is never just whether a model changed. It’s which training choices caused the change, and whether the unwanted part can be separated from the useful part.

The experiments turned up both the expected and the weird. Preference training improved Olmo’s general capabilities, but it also made the model more willing to comply with harmful requests, including prompts wrapped in fictional or hypothetical framing. Goodfire traced part of that regression to specific Dolci examples where the preferred response leaned toward compliance and the rejected response refused. Because Ai2 publishes those individual responses, the researchers could identify the examples, test changes, and see whether the fix reduced the regression without wiping out the gains.

The same approach also exposed an odd behavior no one had reason to track in advance: fan-fiction prompts about characters in a pond, passing gas, and killing nearby fish. Preference training made Olmo more likely to write those stories. Strange, yes. Also useful, because it showed predictive data debugging can surface model quirks before anyone has built a benchmark around them.