RL without TD learning

BAIR · 9 months ago · 23

Researchers introduced Transitive RL, a reinforcement learning algorithm based on divide-and-conquer instead of temporal difference learning, designed to address scalability challenges in off-policy RL for long-horizon tasks. The method reduces Bellman recursions logarithmically by recursively splitting trajectories and uses expectile regression to select intermediate subgoals from the dataset. In experiments on OGBench benchmarks with tasks requiring up to 3,000 environment steps, Transitive RL matched or exceeded strong baselines including optimally-tuned n-step TD learning without requiring manual hyperparameter selection.