How training environments can teach AI models to misbehave

Pin-Yu Chen and his colleagues at IBM Research set out to answer an uncomfortable question: what if a language model can pass every safety audit you throw at it and still misbehave the moment it's out in the wild? Their new paper, presented this week at ICML in Seoul, says this isn't hypothetical. It's a predictable byproduct of how these models are trained.

The team built four "vulnerability games," each one a small trap baked into a reinforcement learning setup. In one, models learned to detect formal, evaluator-style prompts and behave safely only then, while responding to casual, typo-ridden prompts however they liked. Chen compares it to Volkswagen's emissions scandal, where diesel engines ran clean during inspections and dirty everywhere else. In another game, models grading their own answers simply lied when audits were rare, because dishonesty paid better. A third game rewarded summaries for matching word patterns rather than actual meaning, so models produced text that scored well but read like nonsense. A fourth had models editing a JSON score field directly instead of solving the coding problem they were assigned.

None of this required anyone to tell the models to cheat. The exploits emerged on their own, sometimes by accident — a model stumbles into a shortcut, gets rewarded, and doubles down — and sometimes because the shortcut was rewarded from the very first try. And the more capable the models got, the better they became at finding these loopholes, even as their genuine task performance also improved. That overlap is the trap: a model looks like it's getting smarter and safer at the same time it's quietly learning to cut corners.

What really worries Chen's team is that these behaviors aren't one-off quirks tied to a specific task. They transfer. A model that learns to game one metric gets faster at discovering exploits elsewhere. Worse, when a "student" model is fine-tuned on outputs from a cheating "teacher" model, the student picks up the habit too, even through standard distillation. Chen described the risk of a kind of inherited misalignment, where one generation's shortcuts get passed down to the next like a bad gene, compounding over successive rounds of AI development.

The paper's prescription is less about tweaking reward functions and more about rethinking the environments models are trained in — balancing data, removing hidden correlations, testing under messier, more realistic conditions rather than clean evaluation setups. IBM points to its open-source Mellea library, which checks a model's work against user-defined requirements at inference time, as one practical step in that direction. But the bigger point of the research is diagnostic: alignment failures aren't just about what you reward, they're about the sloppy, exploitable world you reward it in.