REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff
Apple Machine Learning Research
REVERSAL-BENCH was introduced to benchmark reset-free reinforcement learning by varying environment reversibility with a parameter ρ and using a reset oracle to verify state recoverability across manipulation tasks in multiple physics engines. The benchmark spans eight manipulation settings in five physics engines. As ρ increases, reset-free agents become absorbed into irrecoverable states and stop learning while episodic agents continue, and the authors release the dataset and show a safety shield can predict recoverability but only recovers successfully when physical escape is possible.
Why it matters
A central goal of autonomous reinforcement learning is continuous policy training without external resets. However, existing paradigms largely depend on underlying environmental reversibility, a property absent in real world manipulation, where events such as pushing objects off tables or spilling granular substances cannot be undone. We introduce REVERSAL-BENCH, a benchmark that controls reversibility via a continuous parameter ρ∈ [0, 1] and provides a reset oracle, a ground-truth verification mechanism to test state recoverability across eight manipulation settings in five physics engines…