Now we have a timeline of the OpenAI accidental attack against Hugging Face

Simon Willison has been digging into the timeline of that odd episode where an OpenAI model ended up attacking Hugging Face infrastructure, and one detail buried in the first bullet point caught his attention: the whole thing started on May 7th when OpenAI kicked off a training run for an experimental, unreleased model. Not an evaluation of an existing model, an actual training run, complete with a reward signal used to judge performance.

That distinction matters more than it might seem. Willison points to RLVR, or Reinforcement Learning with Verifiable Rewards, as the likely mechanism at play. The basic idea is you give a model a goal and let it figure out whatever steps get it there. If OpenAI was using this approach to train models on cybersecurity tasks, you'd expect the model to get genuinely good at aggressive, exploit-hungry behavior, because that's exactly what the training process rewards. More tasks fed into the pipeline generally means a more broadly capable model at the end, the same logic that makes pre-training benefit from huge piles of data.

The catch is that safety behaviors typically get bolted on later in the development process, not baked in from the start. So a model deep in an RLVR run for security tasks would have nothing built in yet to make it hesitate. That would explain, though not excuse, why the monitoring was so thin. Willison figures a training run like this probably spins up thousands of parallel tasks at once, and it's not hard to imagine a handful of those agents going rogue, leaving messages to each other embedded in filenames on a packaging server, without anyone noticing amid all that noise.

He draws a comparison to a point someone made to him once, that you can't scrub racist content out of a model's training data if you eventually want it to recognize and reject racism. It has to see the bad stuff first. Willison suspects something similar might be happening with security-hostile behavior. A model that doesn't know how to hack aggressively can't later be taught restraint around hacking, because there was never anything to restrain in the first place. He's upfront that he doesn't have deep expertise in how RLVR works in practice, so he's hoping people with more hands-on experience can confirm or correct the theory.