Non-Stationary Reward Function Challenge
Deep Reinforcement Learning from Human Preferences — introduced
The observation that because the reward predictor is repeatedly updated during training, the policy is effectively solving an RL problem with a non-stationary reward function; this motivated favoring policy-gradient methods robust to a changing reward and increasing the entropy bonus (particularly for TRPO) to maintain adequate exploration.
A policy chasing a reward function under continuous revision is solving a fundamentally different problem than one chasing a fixed target, and the difference shows up in which algorithms even hold up.
The page names the source of the moving target, then the two ways the paper responds to it, choice of algorithm and a raised entropy bonus, before closing on how the identical problem resurfaces later whenever a language-model reward model is itself refit partway through training.
A moving target, by design
Because the reward predictor is repeatedly refit on new comparisons throughout training rather than fixed once, the policy is, at every point, solving a reinforcement-learning problem whose reward function keeps changing. The paper names this directly: the reward function "may be non-stationary, which leads us to prefer methods which are robust to changes in the reward function" (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4). This is the direct consequence of iterated online training — the choice to keep collecting and fitting on comparisons throughout policy training rather than freezing the predictor after an initial batch, as the "no online queries" ablation does instead.
Two responses: choice of algorithm, choice of entropy bonus
The paper responds in two distinct ways. First, non-stationarity is why the authors "focus on policy gradient methods," settling on A2C for Atari and TRPO for the MuJoCo tasks rather than value-based methods tuned for a fixed reward (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4). Second, within TRPO specifically, the paper raises the entropy bonus above its standard setting — the only hyperparameter changed anywhere in the simulated-robotics experiments — because TRPO's trust region normally guarantees adequate exploration, and that guarantee weakens when the reward driving it is itself being rewritten during training (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14). A2C required no equivalent adjustment.
The same problem, resurfacing under iteration
InstructGPT's PPO fine-tuning stage looks stationary at first glance, since a single reward model is trained once on a fixed comparison dataset before policy optimization begins. But the paper notes that comparison collection and reward-model training can themselves be iterated continuously, with more comparisons gathered on the current best policy to train a new RM and then a new policy in turn (instructgpt, §"3.1 High-level methodology", p. 6) — and whenever that iteration is used, the same non-stationary-target problem this concept names reappears: a policy optimizing against a reward model that is itself about to be replaced by a newer one fit on the policy's own more recent behavior.