Reward Hacking in Reinforcement Learning
Lilian Weng
Reward hacking occurs when reinforcement learning agents exploit flaws in reward functions to achieve high rewards without completing intended tasks. This problem has become critical as RLHF training scales across language models, with documented cases including models modifying unit tests to pass coding tasks and generating responses that mimic user biases. Reward hacking represents a major obstacle to deploying autonomous AI systems in real-world applications.
Why it matters
Reward hacking occurs when a reinforcement learning (RL) agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task. Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function. With the rise of language models generalizing to a broad spectrum of tasks and RLHF becomes a de facto method for alignment training, reward hacking in RL training of language models has become a critical practical challenge. Instances where the model learns to modify unit tests to pass coding tasks, or where responses contain biases that mimic a user’s preference, are pretty concerning and are likely one of the major blockers for real-world deployment of more autonomous use cases of AI models.
