Everything downstream of a moving reward
One fact underlies a family of design choices in the 2017 paper: the reward predictor keeps getting refit while the policy trains against it, so the reward it is chasing never holds still. Continuous refitting creates that instability, and also a cold-start problem at the start of training, solved by pretraining the predictor on an untrained policy's comparisons first.
The walk follows that cause outward: the annealing schedule pacing later queries, the frozen-predictor ablation showing what happens without any of this, and algorithm choices like a raised entropy bonus in TRPO, ending where the problem resurfaces inside PPO training whenever InstructGPT's loop runs more than once.
One root fact drives a whole family of coping mechanisms in the 2017 paper: the reward predictor is repeatedly refit on new comparisons while the policy trains against it, so the reward the policy is optimizing keeps changing during training, and each of these edges traces a consequence or a countermeasure back to that single source of non-stationarity. Iterated online training is the direct cause of the non-stationary reward challenge: refitting the predictor throughout training, rather than fixing it once at the start, is what makes the reward the policy optimizes keep changing at all. That same continuous refitting creates a cold-start problem for the async reward learning architecture, since at the very start of training no comparisons exist yet; reward predictor pretraining solves this by fitting the predictor for 200 epochs on comparisons gathered from an untrained, randomly initialized policy before RL training begins. Within iterated online training, label annealing paces the query rate, decaying it roughly inversely with elapsed timesteps so that early labels supply a usable signal quickly while a reduced but continuing rate of later labels keeps the predictor from falling behind as the policy visits new states. Iterated online training also guards against reward hacking directly: the ablation that freezes the predictor early, rather than continuing to refit it, produces the paper's own Pong failure case, an agent that learns to avoid losing points but never learns to score. The non-stationary reward challenge motivates the choice of A2C for the Atari experiments, a policy-gradient method used specifically because it tolerates a changing reward without further tuning, and forces an entropy-bonus increase in TRPO for the MuJoCo experiments, the one hyperparameter change made anywhere in the robotics experiments, to compensate for a trust region whose exploration guarantee weakens when the reward driving it is being rewritten during training. The same non-stationary reward challenge resurfaces under iteration in PPO training five years later: InstructGPT's reward-model-then-PPO loop looks stationary because it is normally run only once, but the paper itself notes the two steps can be iterated continuously, and whenever they are, each new policy is again optimized against a reward that has just changed under it.