Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models

arXiv cs.AI · 18 hours ago

Researchers developed DROPJ, a method that learns a world model simulator from real-world trajectories, then uses human preferences and safety justifications on simulated trajectory pairs to train a reward model for safe agent deployment. The approach reduced computational training costs and improved deployment performance compared to alternative strategies in real-user experiments. The addition of safety justifications with preferences enhanced safety prioritization during agent deployment in safety-critical environments.