Reward Predictor Pretraining — solves the cold start problem of → Asynchronous Reward-Learning Architecture

The asynchronous architecture's three loops only work once the reward predictor has something to say about the policy's behavior, and at the very start of training it has nothing: no policy trajectories have been compared yet. Reward predictor pretraining solves this specific bootstrapping gap in the Atari experiments by fitting the predictor for 200 epochs on an initial batch of comparisons drawn from an untrained, randomly-initialized policy before RL training even begins (deep-rl-human-prefs, §"A Experimental Details", p. 14). Without this step, the paper notes, the policy risks irreversibly converging on a bad behavior while being guided by a predictor that has barely been trained at all. So the extension is not a generic 'more pretraining is better' trick; it exists because the architecture's three concurrent processes have a specific ordering dependency at initialization that a synchronous, single-pass training setup would not face.