Why Do AI Systems Misbehave?
CSET Georgetown 2 months ago 37
AI misbehavior results from interactions among five components: training data quality, the model's training objective, neural architecture design, system-level guardrails and sampling strategies, and conversation context. Data biases account for a large portion of failures—a medical imaging model once relied on hospital watermarks rather than anatomy, and a horse-to-zebra converter added zebra stripes to rider clothing. Identifying which component causes misbehavior enables targeted fixes, though improvements in one area can sometimes create vulnerabilities elsewhere.