Exploration Strategies in Deep Reinforcement Learning
Lilian Weng
This article discusses exploration strategies in deep reinforcement learning, focusing on how RL agents balance finding optimal solutions quickly while avoiding premature commitment to suboptimal policies. The article covers classic exploration methods like epsilon-greedy and Boltzmann exploration, then details modern deep RL approaches including count-based exploration using density models and hashing techniques with SimHash for high-dimensional states. The strategies aim to address hard-exploration problems in sparse reward environments and the noisy-TV problem where agents become distracted by irrelevant novel experiences.
Why it matters
[Updated on 2020-06-17: Add “exploration via disagreement” in the “Forward Dynamics” section. Exploitation versus exploration is a critical topic in Reinforcement Learning. We’d like the RL agent to find the best solution as fast as possible. However, in the meantime, committing to solutions too quickly without enough exploration sounds pretty bad, as it could lead to local minima or total failure. Modern RL algorithms that optimize for the best returns can achieve good exploitation quite efficiently, while exploration remains more like an open topic.
