Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners

arXiv cs.AI · 18 hours ago

Researchers conducted an empirical study of actor-critic reinforcement learning algorithms by running over 33,000 experiments on a water treatment control task to evaluate how different design choices affect reliability and hyperparameter sensitivity. Key findings showed that common defaults like Gaussian action distributions with pathwise gradient estimators performed poorly, while bounded action distributions with adaptive update schedules demonstrated robustness across various settings. The results provide practitioners in scientific and engineering domains with guidance for selecting actor-critic components when deploying these algorithms for real-world control applications.