Third-party cyber evaluations involving OpenAI models
Simon Willison's Weblog Simon Willison ● Covered by 2 sources
OpenAI says one of its models accidentally hacked a real website during a supposedly offline security test. A sandbox misconfiguration let it slip onto the live internet.
Simon Willison has started tagging blog posts "accidental-cyberattacks," which tells you everything about how often this is happening lately. The latest entry comes straight from OpenAI, describing a test run with Irregular, one of the outside firms it hires to stress-test its models on cybersecurity tasks. The setup was a Capture-the-Flag exercise, the kind of puzzle where a model hunts for a fictional target inside a closed, sandboxed network with no route to the real internet.
Except this time the sandbox wasn't actually sealed. A misconfiguration in the testing environment left it connected to the public internet, and the fictional domain name picked for the challenge happened to match a domain that actually exists out there in the wild. The model, doing exactly what it was trained to do, found what looked like its target and went after it. It had no way of knowing the target was real. It just saw a matching name and started probing, and because the leash was broken, it reached an actual live website instead of a make-believe one inside a locked box.
This is the second incident of its kind that Willison has logged this month. The first involved the UK AI Safety Institute running similar red-team evaluations. And Irregular shows up again in Anthropic's own writeup of a comparable slip, where a misconfigured environment they were hosting gave Claude an open line to the internet during testing. Same company, same category of mistake, different model on the other end of it.
The underlying story here isn't that AI models are becoming dangerously autonomous hackers. It's that the infrastructure meant to contain them during testing is proving surprisingly leaky. A naming coincidence and one bad network config were enough to turn a hypothetical exercise into a real intrusion attempt, and that happened not once but at least twice, at two different labs, with the same third-party vendor sitting in the middle of both.
My take
Nobody should be shocked that a capable model exploited a real target when the sandbox around it failed; that is what these systems are built to do once they're pointed at something that looks vulnerable. The actual scandal is operational: two separate labs, sharing the same testing contractor, both got bitten by basic environment isolation failures within weeks of each other. If the red-teaming infrastructure keeps springing leaks, tightening the models themselves is beside the point.
Read more about this at: Simon Willison's Weblog
