Keeping the reward model on-distribution — shares non stationarity with → Policy optimization under a learned, changing reward

These two themes are the same non-stationarity viewed from either side of the training loop, and each narrative describes its own half without naming the other as the reason it exists. On the reward-model side, online training, ensembling, regularization, and label annealing exist because the predictor is chasing a policy that keeps visiting new states, an unlabeled region the fixed early fit never saw (deep-rl-human-prefs, §"2.2 Our Method", p. 4). On the policy side, the paper picks A2C and TRPO specifically because policy-gradient methods tolerate a reward that keeps moving, and it raises TRPO's entropy bonus above standard settings for exactly this reason: "TRPO relies on the trust region to ensure adequate exploration, which can lead to inadequate exploration if the reward function is changing" (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 5). Neither fix alone would hold: an optimizer built to tolerate a stale reward gains nothing if the reward model never refreshes, and a constantly-refreshed reward model gains nothing if the optimizer's own exploration collapses before the refreshed signal reaches the states that need it. The two themes are better read as a single design decision split across two pieces of the architecture, one absorbing the non-stationarity by adapting the reward, the other by choosing an optimizer built to survive it.