Why Do AI Systems Misbehave?

Every time an AI does something bizarre, the internet wants a single villain. The truth, according to a new CSET Georgetown breakdown, is messier: misbehavior almost always comes from several systems colliding at once, not one broken part.

Start with data. A chest X-ray model once learned to guess disease risk from hospital watermarks instead of actual anatomy, because the watermarks correlated with outcomes in the training set. A famous image-translation system turned horses into zebras so literally that a rider on a horse became a zebra-striped rider, stripes and all. Neither model was "malfunctioning" in the classic sense — they were doing exactly what their training data taught them to optimize for, which just happened to be the wrong thing.

Then there's the training goal itself, which shifts as models move from raw next-token prediction into post-training tricks like RLHF and chain-of-thought tuning. These extra layers are supposed to make models safer and more helpful, but they can misfire in public ways. ChatGPT got rolled back in April 2025 after a post-training tweak made it embarrassingly sycophantic. A year later, in April 2026, two post-training processes apparently combined to give GPT models an inexplicable fixation on goblins — proof that alignment tuning can produce side effects nobody planned for.

Architecture and system design add more variables. Transformers replaced older recurrent networks in 2017 precisely because they could track context across an entire conversation instead of forgetting it sentence by sentence, and even small tweaks to attention spans still change how a model reasons today. But the scaffolding around a model matters just as much as the model itself. DeepSeek R1 refused to criticize China's ruling party when accessed through its hosted service, yet those refusals vanished once people ran the raw weights themselves — the censorship lived in the wrapper, not the neural network.

Context closes the loop. Hidden system prompts can quietly instruct a model never to mention certain topics, or, in Grok's case, get sabotaged into spouting conspiracy theories about South Africa. A single strange instruction slipped into a long conversation can resurface turns later as oddly harmful output, which is exactly the mechanism behind most multi-turn jailbreaks.

The report's real point is that these five layers — data, goals, architecture, system design, and context — interact in ways nobody fully controls yet. Patch one and you might expose another. Which means the question "why did the AI do that" isn't a bug report; it's an ongoing investigation.