Asynchronous Reward-Learning Architecture
Deep Reinforcement Learning from Human Preferences — introduced
The paper's overall training architecture in which a policy, a reward predictor, and a human-preference-collection process run as three concurrent, asynchronously-coupled loops: the policy acts to generate trajectories, segment pairs are sent to a human for comparison, and the reward predictor is retrained on the growing preference database, with each process's outputs continuously feeding the next.
Training a policy, collecting human judgments, and refitting a reward model are three separate jobs, and doing them one after another would leave long stretches where nothing useful happens. The paper instead runs all three at once, continuously, each one feeding the next.
The page lays out the three loops and how their outputs pass between each other, then the problems concurrency itself creates, a cold start before any comparisons exist, a target that keeps moving, before closing on how later systems keep this design's two-stage logic while dropping its concurrency.
Three processes, one continuous loop
The paper's method maintains a policy $\pi$ and a reward function estimate $\hat r$, each updated by three processes that run concurrently rather than in sequence: the policy acts in the environment to produce trajectories, pairs of segments drawn from those trajectories are sent to a human for comparison, and the reward predictor is refit by supervised learning on the growing set of recorded comparisons (deep-rl-human-prefs, §"2.2 Our Method", p. 4). Trajectories flow from the first process to the second, comparisons flow from the second to the third, and updated parameters for $\hat r$ flow back to the first — "these processes run asynchronously" (deep-rl-human-prefs, §"2.2 Our Method", p. 4), so training never pauses to wait on a finished dataset the way a supervised pipeline would. Figure 1 depicts exactly this cycle, and the paper's introduction frames the underlying idea plainly: "learn a reward function from human feedback and then to optimize that reward function" (deep-rl-human-prefs, §"1 Introduction", p. 2).
Coordinating three moving parts
Running three loops concurrently creates problems a single-pass pipeline never faces, and several of this concept's children exist to fix one of them. Reward Predictor Pretraining solves the bootstrapping gap at the very start of training, when no trajectories have yet been compared and the reward predictor has nothing to say about behavior; in the Atari experiments it pretrains the predictor for 200 epochs on comparisons drawn from an untrained policy before RL begins (deep-rl-human-prefs, §"A Experimental Details", p. 14). Label Annealing paces how fast new comparisons are requested as training proceeds, and the Non-Stationary Reward Function Challenge names a direct consequence of continually refitting the predictor: the policy is chasing a reward function that keeps changing under it. Preference Elicitation Protocol is the mechanism that turns trajectories into recorded comparisons in the first place. Together these are why the architecture belongs to the Keeping the reward model on-distribution theme as much as to the paper's headline method.
Naming a paradigm, and losing its concurrency later
The architecture is what the paper points to when it distinguishes its contribution from prior preference-learning work: learning a reward function from human feedback and then optimizing it is the general recipe, but the asynchronous, three-process design is what makes that recipe executable at deep-RL scale rather than merely stated. This is the concrete shape the paper gives to what would later be named RLHF. Later systems keep the recipe's two-stage structure — learn a reward, then optimize it — but drop the concurrency: InstructGPT's Reward model (RM) is fit on a fixed, already-collected comparison dataset rather than being continuously refit against a policy that is simultaneously generating new trajectories, trading the three-loop coordination problem this paper solves for a simpler but less adaptive two-stage pipeline. That lineage is also why the concept sits in the From an open problem to reward models to AI feedback theme.