Reward Predictor Pretraining
Deep Reinforcement Learning from Human Preferences — introduced
In the Atari experiments, the reward predictor is pretrained for 200 epochs on an initial batch of comparisons (drawn from an untrained, randomly-initialized policy) before RL training begins, reducing the risk of the policy irreversibly converging on a bad behavior guided by an untrained predictor.
At the very start of training, a reward model has nothing yet to say about an agent's behavior, and letting a policy optimize against it immediately can do real damage. The paper buys itself a head start before that risk sets in.
The page covers what the pretraining phase actually does, using comparisons from a policy that hasn't learned anything yet, then why an untrained predictor at the very start of training is a risk worth guarding against, and closes on how this fix is confined to only one of the paper's two experimental domains.
Priming the predictor before RL begins
In the Atari experiments, before RL training starts, the paper compares a number of trajectory segments drawn from rollouts of an untrained, randomly initialized policy, and pretrains the reward predictor on this initial batch of comparisons for 200 epochs (deep-rl-human-prefs, §"A Experimental Details", p. 14). Only after this pretraining does the policy begin optimizing against the predictor.
Why the cold start matters
The asynchronous architecture's three loops only work once the reward predictor has something to say about the policy's behavior, and at the very start of training it has nothing: no trajectories have yet been compared. Pretraining exists specifically "to reduce the likelihood of irreversibly learning a bad policy based on an untrained predictor" (deep-rl-human-prefs, §"A Experimental Details", p. 14) — an untrained predictor could steer the policy toward behavior it can never escape, since the RL loop and the labeling loop run concurrently rather than waiting on each other. The fix is confined to the Atari experiments in this paper; the MuJoCo tasks instead draw only 25% of their initial comparisons from an untrained policy, without a dedicated pretraining phase.