Reinforcement learning from human feedback (RLHF)
Training language models to follow instructions with human feedback — refined; Constitutional AI: Harmlessness from AI Feedback — inherited; Deep Reinforcement Learning from Human Preferences — introduced; Steering Llama 2 via Contrastive Activation Addition — inherited
A training paradigm that fits a reward model to human preference judgments and then optimizes a policy against that learned reward with a standard RL algorithm, rather than hand-specifying a reward function. Introduced in 2017 for simulated robotics and Atari games, it was scaled in 2022 to align large language models to a broad, open-ended distribution of natural-language instructions.
Hand-writing a reward function works fine until the behavior you want is easy to recognize but hard to specify in code. Reinforcement learning from human feedback sidesteps that wall by having people compare outputs directly and letting a learned model absorb their judgment instead of asking anyone to write the rule down.
The page traces the paradigm to its 2017 introduction in simulated robotics and Atari, covers the comparison-based fitting and the three concurrent training loops that made it workable at scale, then follows the same two-stage recipe into InstructGPT (2022), which scales it to language, and closes on its place in a longer scalable-oversight lineage.
Introduced in 2017: learning a reward function instead of hand-specifying one
RLHF names the general paradigm the 2017 paper introduces: rather than hand-specify a reward function, "fit a reward function to the human's preferences while simultaneously training a policy to optimize the current predicted reward function" (deep-rl-human-prefs, §"2.2 Our Method", p. 4). The paper's stated motivation anticipates the scalable-oversight problem Concrete Problems had named a year earlier: using human feedback directly as a reward signal is prohibitively expensive for RL systems that require hundreds or thousands of hours of experience, so the paper sets four desiderata for its solution: "1. enables us to solve tasks for which we can only recognize the desired behavior, but not necessarily demonstrate it, 2. allows agents to be taught by non-expert users, 3. scales to large problems, and 4. is economical with user feedback" (deep-rl-human-prefs, §"1 Introduction", p. 2). It reports meeting the last of these by roughly three orders of magnitude, solving benchmark tasks while providing feedback on less than 1% of the agent's interactions with the environment (deep-rl-human-prefs, §"Abstract", p. 1).
The mechanics: comparisons, Bradley-Terry fitting, asynchronous training
Concretely, the paradigm asks a human to compare short video clips of trajectory segments rather than supply an absolute score, then fits a reward predictor by treating summed predicted reward as a latent score in the Bradley-Terry pairwise-preference model (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). What makes the paradigm executable rather than merely stated is its architecture: the policy, the human-comparison process, and the reward predictor run as three concurrent, never-terminating loops, with trajectories flowing to comparisons, comparisons flowing to the predictor, and the predictor's parameters flowing back to the policy (Asynchronous Reward-Learning Architecture — gives concrete shape to → Reinforcement learning from human feedback (RLHF); deep-rl-human-prefs, §"2.2 Our Method", p. 4). This is also the origin of what the corpus later calls Iterated (Online) Training: refitting the predictor continuously on labels gathered throughout training, rather than once on an early fixed batch, because a static predictor is what the paper's own ablation shows produces reward-hacking-like behavior (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 9).
Scaled to language models: InstructGPT's three-step pipeline
InstructGPT extends this established paradigm, for the first time, to a broad, open-ended distribution of natural-language instructions spanning generation, question answering, brainstorming, and more, rather than the narrow domains, simulated robotic control and Atari in the 2017 paper, stylistic continuation and summarization in Ziegler et al. (2019) and Stiennon et al. (2020), that RLHF had previously been applied to (instructgpt, §"1 Introduction", p. 2; §"3.1 High-level methodology", p. 6). In this paper, RLHF names the whole three-step procedure gathered under the InstructGPT's three-step training pipeline theme: Supervised fine-tuning (SFT) on labeler demonstrations produces the Supervised fine-tuned (SFT) model; a Reward model (RM) is trained on labeler rankings to predict preferred outputs; and PPO fine-tuning stage (RLHF policy optimization) then optimizes the SFT model against that learned reward. The Reward model (RM) — anchors the reward signal of → Reinforcement learning from human feedback (RLHF) edge notes the reward model's architecture is not incidental: it is built by replacing the SFT model's unembedding layer with a scalar head, so it starts from the same weights as the policy it will later score, and its own weights in turn initialize the PPO PPO value function.
Its place in a scalable-oversight lineage, and Constitutional AI's AI-feedback variant
Within the From an open problem to reward models to AI feedback theme, RLHF is the first working answer to the open problem Concrete Problems named in 2016: safe behavior requires evaluating an objective too costly to check on every example, and a learned reward model lets a cheap, dense proxy stand in for genuinely expensive human judgment across an entire training distribution — an answer the 2017 paper builds directly and InstructGPT scales up six years later. The Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF) edge traces this lineage one step further into Constitutional AI, whose RL from AI Feedback keeps "the remainder of the training pipeline...exactly the same as RLHF" (constitutional-ai, §"4.1 Method", p. 10) and changes only who answers the comparison question for harmlessness, an AI feedback model instead of a human labeler; RLHF is accordingly inherited by Constitutional AI as an established technique it extends and partially replaces (constitutional-ai, §"1 Introduction", p. 2; §"5 Related Work", p. 14). In the graph's hierarchy, Helpful RLHF Model and HH RLHF Model are Constitutional AI's own applications of this technique, sitting as children of this concept outside the present paper's scope.
2023: activation steering operates on top of the finished RLHF stack
Steering Llama 2 via Contrastive Activation Addition (2023) is the corpus's first paper to treat RLHF as a finished, fixed substrate to be probed rather than a training-time target to be modified. Every experiment in Contrastive Activation Addition (CAA) runs on "Llama 2 7B and 13B Chat...that have been trained using RLHF for safety and to follow human instructions in a chat format" (contrastive-activation-addition, §"3 Method", p. 3); the paper is never tested on a raw pretrained model, and it frames the whole study as an attempt "to study the interaction between RLHF/finetuning techniques and activation engineering, building on top of the existing body of research on pretrained models" (contrastive-activation-addition, §"1 Introduction", p. 2). That framing matters for how to read every later result: when a Steering vector moves a behavior beyond what System-prompting or Supervised fine-tuning (SFT) alone achieve, it is composing with an RLHF-trained network rather than steering a blank slate, and it still finds room to move — evidence that a finished RLHF run leaves headroom a post-hoc intervention can exploit rather than saturating the space of expressible behavior.
Base versus chat: the corpus's first internal measurement of what RLHF changed
Where the rest of this page studies RLHF from the outside, through the outputs it changes, Base vs. RLHF-chat model representation comparison is the corpus's first attempt to look at what RLHF leaves alone on the inside. Steering vectors are generated separately from Llama 2 7B base and Llama 2 7B Chat activations and compared layer by layer with cosine similarity; similarity decays with layer distance as expected, except for a distinct peak between layers 7 and 15, where the base and chat representations stay unusually close — "this surprising trend indicates that RLHF has a smaller effect on how information is represented between layers 7 and 15" (contrastive-activation-addition, §"8.3 Comparing representations between base and chat models", p. 7). Vectors extracted from the base model's activations then transfer significant steering effect onto the RLHF-tuned chat model, especially between layers 10 and 15 (contrastive-activation-addition, §"8.3 Comparing representations between base and chat models", p. 9). Read together, this is the first direct evidence in the corpus about the mechanism by which RLHF training changes a network: it appears to re-weight which behaviors get expressed downstream while leaving the mid-network representation of those behaviors largely intact, rather than rewriting that representation outright.