Reward model (RM)
Training language models to follow instructions with human feedback — refined; Constitutional AI: Harmlessness from AI Feedback — inherited; Deep Reinforcement Learning from Human Preferences — introduced
A neural network trained via supervised learning to predict a scalar reward consistent with human pairwise preferences over behavior, used as a stand-in for a reward signal that is hard to specify directly. Introduced in 2017 for robotics and Atari control, then scaled in 2022 to steer large language models trained with reinforcement learning from human feedback.
Some goals are easy to recognize and hard to write down as a rule, leaving a gap no hand-crafted reward function can fill on its own. A reward model is the paper's answer: a network trained to predict which of two outputs a person would prefer, standing in for the reward nobody could specify directly.
The page opens with the 2017 paper's original predictor, fit from trajectory-segment comparisons rather than absolute scores and refit continuously alongside the policy it scores, then follows the same architecture into InstructGPT (2022), which scales it to language, and closes with Constitutional AI (2022), which feeds the identical recipe AI-sourced comparisons instead of human ones.
From a named problem to a working predictor
Concrete Problems in AI Safety names the problem this concept eventually solves — Scalable oversight, the difficulty of ensuring safe behavior when the true objective function is too costly to evaluate on every training example — but supplies no working system built to answer it (concrete-problems, §"5 Scalable Oversight", p. 11). The first working instance is the reward predictor $\hat r$ of deep-rl-human-prefs: a neural network trained via supervised learning to fit a database of human pairwise preferences over short trajectory-segment clips, then substituted directly for the environment's true reward inside an otherwise ordinary RL loop (deep-rl-human-prefs, §"2.2 Our Method", p. 4). This predictor is the same concept later called the reward model (RM) by InstructGPT and the preference model (PM) by Constitutional AI — three names for one architecture-and-training-recipe lineage, with the 2017 paper as the introducing anchor and InstructGPT and Constitutional AI as, respectively, its language-model scaling and its AI-feedback variant. Concrete Problems' own child concept Supervised reward learning is the informal 2016 sketch of exactly this idea, predicting reward on unlabeled data in place of a ground-truth signal, and Semi-supervised or active reward learning is the sibling proposal whose first deep-RL implementation is this same 2017 paper's query-selection mechanism.
How the 2017 predictor is fit: comparisons, not scores
The reward predictor is never given an absolute number to regress against. It is fit by treating summed predicted reward over a trajectory segment as a latent score in the Bradley-Terry pairwise-comparison model, so the probability the predictor assigns to a human preferring one segment over another is a softmax over the two summed rewards, and training minimizes the cross-entropy between that prediction and the recorded human choice (Bradley-Terry Model — supplies the loss for → Reward model (RM); deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). The full mathematical form is Reward model pairwise comparison loss; three refinements measurably improve it: an ensemble of independently-normalized predictors trained on bootstrap-resampled comparisons and then averaged; an adaptive MATH0 penalty tuned against a held-out $1/e$ slice of the comparison database so validation loss stays between 1.1 and 1.5 times training loss; and a 10%-random-response assumption that keeps the model from demanding implausibly extreme confidence when a human simply errs (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). Because the loss only ever sees differences between rewards, its raw output has no fixed zero or scale (Reward model pairwise comparison loss — under determines the scale of → Reward model (RM)); Reward Prediction Normalization fixes this by rescaling the predictor's output to zero mean and constant standard deviation before it reaches the RL algorithm (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 5). Eliciting comparisons rather than absolute scores was itself a deliberate choice, Comparisons vs. Absolute Scores (Target Ablation), because the paper found comparisons easier for humans to give consistently, especially on continuous-control and qualitative tasks (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 10). The reward function actually built is also narrower than the paper's own general formulation admits: it is defined over a single observation-action pair, widened for Atari only to a fixed stack of the preceding 4 observations, not an agent's full history, which is why Recurrent Reward Model (Proposed Extension) is flagged as a proposed but unbuilt generalization (deep-rl-human-prefs, §"2.1 Setting and Goal", p. 4). Even the network architecture reflects headroom the tasks at hand didn't need: the MuJoCo reward predictor is a full two-layer network even though the true reward functions there are simple polynomials of the input features, a deliberately overbuilt choice made to generalize later to reward functions that are not so simple (Reward model (RM) — is overbuilt in anticipation of → Novel Behavior Training; deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14).
Trained asynchronously, alongside the policy it scores
The reward predictor is never fit once and frozen; it is one of three concurrent, indefinitely-running loops that make up the Asynchronous Reward-Learning Architecture: the policy interacts with the environment to produce trajectories, pairs of segments from those trajectories go to a human for comparison, and the predictor is continually refit by supervised learning to the comparisons collected so far, with trajectories, comparisons, and parameters each flowing to the next stage without any stage ever stopping (deep-rl-human-prefs, §"2.2 Our Method", p. 4). This is also the origin of what the corpus later names Iterated (Online) Training: refitting the predictor continuously on labels gathered throughout training, rather than once on an early fixed batch, is exactly what the paper's own ablation shows matters. Trading that concurrency away for a fixed early dataset is what the "no online queries" ablation does, and it is what produces the paper's own worked example of reward hacking rather than a stable policy (Asynchronous Reward-Learning Architecture — trades its concurrency for a fixed dataset in → Reward model (RM); deep-rl-human-prefs, §"3.3 Ablation Studies", p. 9). The general paradigm this architecture instantiates, learning a reward model from preferences and optimizing it with a standard RL algorithm rather than hand-specifying a reward function, is what the corpus calls Reinforcement learning from human feedback (RLHF); the reward model is not a free-standing scorer bolted onto that paradigm, since RLHF's structure is dictated by how the reward predictor is trained and reused (Reward model (RM) — anchors the reward signal of → Reinforcement learning from human feedback (RLHF)).
Scaled to language models: InstructGPT's reward model
InstructGPT scales the same recipe, comparisons in, scalar reward out, fit by a ranking loss, to a wholly different domain and a much larger network. Its reward model is built by taking the trained SFT model and replacing its final unembedding layer with a scalar output head, so it starts from the same weights as the policy it will later score (instructgpt, §"3.5 Models", p. 8). It is trained on labeler-ranked completions ($K=4$ to $9$ per prompt) using the K-choose-2 single-batch comparison training trick to avoid overfitting on correlated comparisons drawn from the same ranking. Because the underlying loss is still additive-shift invariant, exactly the scale problem normalization solved in 2017, InstructGPT applies the identical fix in its own domain: outputs are normalized so labeler demonstrations score a mean of 0 before RL begins (instructgpt, §"3.5 Models", p. 9). The RM's weights also initialize the PPO value function, so the same learned preference judgment appears twice in the RL loop, once as the reward being chased and once as the critic estimating future value (The mechanics that make PPO fine-tuning work — reveals a critic reward coupling inside → InstructGPT's three-step training pipeline). Because 175B RM training proved unstable, every InstructGPT policy, including the 175B one, is trained against a single 6B reward model, supervision from a judge nearly 30 times smaller than the policy it steers (Reward model (RM) — supervises from below → PPO fine-tuning stage (RLHF policy optimization); instructgpt, §"C.2 Details of RM training", p. 41). The RM reaches backward in the pipeline too: the final SFT checkpoint is chosen by its score under the reward model rather than by validation loss (Reward model (RM) — back selects the checkpoint of → Supervised fine-tuned (SFT) model; instructgpt, §"3.5 Models", p. 8). Its imperfection as a proxy is not left unaddressed: a per-token Per-token KL penalty from the SFT model against the SFT model, not the RM itself, keeps PPO from over-optimizing this imperfect signal, the specific countermeasure engineered against Reward model over-optimization.
Constitutional AI: the same recipe, AI-sourced labels
Constitutional AI inherits this architecture and training recipe wholesale and renames it: the same object is called the reward model (RM) by InstructGPT and the preference model (PM) by Constitutional AI (constitutional-ai, §"1.2 The Constitutional AI Approach", p. 5), built "following the process in [Bai et al., 2022]" (constitutional-ai, §"4.1 Method", p. 10), a citation into the very 2017 lineage this page traces directly. Constitutional AI's own reward signal, the Hybrid Human/AI Preference Model, is a child of this same concept: identical architecture, fed a differently-sourced comparison set, human-labeled helpfulness comparisons mixed with AI-labeled harmlessness comparisons from a separate Feedback Model. The connection back to the 2016 problem this concept answers is drawn only in hindsight for the language-model leg of the story: neither the 2017 paper nor InstructGPT ever uses the phrase "scalable oversight," and the match is made explicit only once Constitutional AI revives the term under the label "Scaling Supervision," stating that RLHF "has already taken a step in the direction of scaled supervision, since the reward signal in RL actually comes from an AI preference model (PM) rather than from immediate human oversight" (Scalable oversight — is retroactively operationalized by → Reward model (RM); constitutional-ai, §"Scaling Supervision", p. 2).