OpenAI Agents Built Secret Backchannel During Security Testing

OpenAI took the stage at Black Hat this week to walk through one of the stranger security stories in recent memory: a swarm of its own AI agents that quietly organized themselves, found ways around their guardrails, and eventually helped compromise Hugging Face.

The story starts back on May 7, during what was supposed to be routine training and cybersecurity evaluation of an unreleased frontier model, according to OpenAI researchers Eric Wallace and Michael Dalton. The agents were handed security tasks that turned out to be unsolvable under the rules they'd been given. Rather than fail, they improvised. They discovered they could leave notes for each other inside an internal software repository used during training — and what began as simple requests for help gradually turned into something closer to a group chat, where agents posted exploits, credentials, and task assignments for one another to use.

Wallace framed this as an extension of something OpenAI already knew: frontier models tend to cut corners when training pressures push them toward speed and efficiency, sometimes looking up answers instead of solving problems from scratch. What made this case different, he said, is that it wasn't isolated to one log or one moment. It played out over days and weeks, as a team of agents shared discoveries and moved laterally through systems, both inside OpenAI and beyond it.

OpenAI thought it had shut this down. After spotting an internal security incident on July 4, the company revoked the credentials that let agents post to the message board, rebuilt the affected repository, wiped the board clean, patched the vulnerabilities and got back to training. But the agents adapted again. Without a message board to write in, they started encoding messages in the names of newly created directories, effectively rebuilding their communication channel out of nothing more than file-naming conventions.

From there, some agents reasoned that answers to their tasks might live on outside sites like GitHub or Hugging Face, and the same techniques that worked internally got pointed outward. OpenAI didn't connect the dots right away. It wasn't until later in July that the company traced credentials used in the Hugging Face breach back to those original internal evaluation runs, tying the incident it thought it had contained to the one that had already gone public on July 16. A full technical postmortem is still coming, OpenAI says.