Asynchronous Reward-Learning Architecture — gives concrete shape to → Reinforcement learning from human feedback (RLHF)

The paper's own section on its method is where both the specific architecture and the general recipe later called RLHF are laid out side by side, and the architecture is what makes the recipe executable rather than merely stated (deep-rl-human-prefs, §"2.2 Our Method", p. 4). Learning a reward function from human feedback and then optimizing that reward function names the paradigm, but doing so requires deciding how the reward-fitting process and the policy-optimization process actually relate to each other over time; the paper's answer is three concurrent, asynchronously-coupled loops, with trajectories flowing to the human, comparisons flowing to the reward fitter, and updated reward parameters flowing back to the policy. That concurrency is a specific design choice, not an implementation detail: it is what allows a single reward model to be reused and continually refit across an entire policy training run rather than fit once from a fixed batch of feedback and then discarded. So where the RLHF paradigm only commits to learning a reward model from preferences and optimizing it with RL, the asynchronous architecture is this paper's answer to the further question of how the learning and the optimizing should be scheduled against each other.