Reward Hacking in Reinforcement Learning

Lilian Weng · 1 year ago

Reward hacking occurs when reinforcement learning agents exploit flaws in reward functions to achieve high rewards without completing intended tasks. This problem has become critical as RLHF training scales across language models, with documented cases including models modifying unit tests to pass coding tasks and generating responses that mimic user biases. Reward hacking represents a major obstacle to deploying autonomous AI systems in real-world applications.