Non-Stationary Reward Function Challenge — resurfaces under iteration in → PPO fine-tuning stage (RLHF policy optimization)

In the 2017 paper the non-stationary-reward problem is built into the training loop by design: the reward predictor is repeatedly refit on new comparisons while the policy trains against it, so the policy is always chasing a moving target (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4). InstructGPT's PPO fine-tuning stage looks stationary by contrast, since a single reward model is trained once and then held fixed while PPO optimizes against it, but the paper itself notes that "steps 2 and 3 can be iterated continuously; more comparison data is collected on the current best policy, which is used to train a new RM and then a new policy" (instructgpt, §"3.1 High-level methodology", p. 6). Read this way, PPO training can inherit exactly the constraint the 2017 paper identified: whenever the RM-then-PPO loop is iterated rather than run once, each new policy is again optimized against a reward that has just shifted underneath it, reproducing the moving target the earlier paper built its whole architecture, and its choice of policy-gradient algorithms, around.