Asynchronous Reward-Learning Architecture — trades its concurrency for a fixed dataset in → Reward model (RM)

The 2017 paper's defining architectural choice is that nothing is ever finished: the policy, the human-comparison process, and the reward predictor run as three concurrent loops for the entire duration of training, with 'trajectories flowing from process (1) to process (2), human comparisons flowing from process (2) to process (3), and parameters for r̂ flowing from process (3) to process (1)' (deep-rl-human-prefs, §"2.2 Our Method", p. 4). InstructGPT's reward model is built by the opposite discipline: it is trained once, to convergence, on a fixed dataset of roughly 33,000 human-ranked comparison sets collected before PPO ever starts, and its weights are then frozen for the whole of RL fine-tuning (instructgpt, §"3.5 Models", p. 8). That difference reshapes what 'the reward model' even means operationally: in 2017 it is a moving target the RL optimizer must tolerate mid-training, while in InstructGPT it is a static artifact fully specified before optimization begins. InstructGPT accepts the risks of a static reward model deliberately — managing policy drift with a per-token KL penalty against the SFT model — rather than reintroducing anything like the 2017 paper's concurrent retraining loop.