Non-Stationary Reward Function Challenge — motivates the choice of → Advantage Actor-Critic (A2C/A3C)

The paper picks its RL algorithms specifically to survive a reward function that keeps moving underneath the policy: because the reward predictor is repeatedly updated as new preference labels arrive, the optimizer is chasing a non-stationary target, which the authors say led them to focus on policy-gradient methods over alternatives tuned for a fixed reward (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4). A2C is the concrete choice that follows from that reasoning for the Atari domain, used essentially off the shelf with standard hyperparameters from prior work reused unchanged, because being a policy-gradient method already supplied the needed robustness without further modification (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4). Unlike the MuJoCo case with TRPO, no compensating hyperparameter change was judged necessary for A2C, suggesting the paper treats the non-stationarity problem as already handled by the choice of algorithm family, not something every policy-gradient method needs extra tuning to survive.