Policy optimization under a learned, changing reward
Train a policy against a reward model rather than a fixed environment signal, and the reward itself starts moving under the policy's feet as the model keeps getting refit. The 2017 paper responds by picking algorithms built to tolerate a moving target: Advantage Actor-Critic for its seven Atari games, Trust Region Policy Optimization for MuJoCo with its entropy bonus deliberately raised, both fed a reward rescaled to a fixed, predictable range.
The walk starts from that moving-target problem, moves through the two algorithms chosen to survive it, past DQN as a comparison point, and ends on PPO training, where InstructGPT (2022) faces the identical problem five years later.
Once a policy is trained against a reward model rather than a fixed environment signal, the reward itself keeps changing underneath it, and this theme is about the RL algorithms built to tolerate that. The non-stationary reward function challenge is the underlying constraint: because the reward predictor is repeatedly refit during training, the policy is solving a moving-target problem, which the 2017 paper addresses by favoring policy-gradient methods and raising the entropy bonus. Advantage Actor-Critic is the algorithm chosen for the paper's seven Atari tasks, run in its synchronous form with standard hyperparameters reused from prior work. Trust Region Policy Optimization is the algorithm chosen for the MuJoCo robotics tasks, with its entropy bonus raised above standard settings, the one hyperparameter the authors adjusted, specifically to restore exploration lost to the trust region under a shifting reward. Deep Q-Network appears as a comparison point rather than a method used here: the paper notes its own Enduro results are comparable to DQN's reported performance. Reward prediction normalization is the shared preprocessing step feeding both optimizers a reward of fixed scale, since a comparison-trained predictor leaves the reward's absolute scale undetermined. The same constraint reappears six years later inside PPO training, the RL stage of InstructGPT (2022), where the policy is likewise optimized against a reward model that shifts as training proceeds.