Iterated (Online) Training — is the cause of → Non-Stationary Reward Function Challenge

The non-stationary reward function challenge is not an incidental property of the reward model; it is the direct consequence of iterated online training. Because comparisons keep arriving and the reward predictor keeps being refit throughout policy training rather than fit once and fixed, the policy is, at every point, optimizing against a target that itself is still moving (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4). The paper treats this as a genuine cost worth paying: the alternative, fitting the predictor only once from an early batch of labels, is exactly the 'no online queries' ablation, and it performs worse because a static predictor only ever captures part of the true reward. Framed this way, the entropy-bonus increase and the preference for policy-gradient methods are not general-purpose robustness choices; they are specific compensations for a non-stationarity that the paper's own choice to keep training the reward model continuously has introduced.