Deep Reinforcement Learning from Human Preferences

Inverse reinforcement learning opens this theme as the oldest and most direct alternative to the 2017 paper's approach: extract a reward function from demonstrations of the desired task, then train an agent against that extracted reward using ordinary reinforcement learning. The paper notes this is inapplicable when the desired behavior is hard for a human to demonstrate in the first place, such as controlling a robot with a non-human morphology.

Cooperative inverse reinforcement learning (Hadfield-Menell et al., 2016) closes this theme by generalizing beyond any single feedback mechanism: it frames a human and an agent as two players cooperating to maximize the human's reward function. The 2017 paper positions its own method as a specific instance of this framework, one in which the human's only available move is to state a preference between two segments.

A trajectory segment is the smallest unit the 2017 paper asks a human to judge: a clip of one to two seconds of observations and actions, short enough to show quickly but long enough to carry more information than a single state. Every mechanism in this theme exists to turn judgments about pairs of these clips into a usable reward signal.

The preference elicitation protocol is the procedure that actually collects those judgments: a human overseer watches two rendered trajectory-segment clips side by side and records a preference, a tie, or an incomparable response into a growing database. This raw preference data is what the next step has to convert into a scale a reinforcement learning algorithm can optimize.

The Bradley-Terry model, borrowed from 1952 paired-comparison statistics, supplies that conversion: it treats each segment's predicted summed reward as a latent score and passes the difference through a softmax to predict the probability a human prefers one segment over the other. Fitting this softmax to the protocol's recorded preferences is what actually trains the reward predictor.

Comparisons vs. absolute scores is the paper's own stress test of the choice underlying everything so far: an ablation that replaces the comparison-based cross-entropy loss with direct mean-squared-error regression of the reward predictor to an oracle's true segment reward. The paper finds humans give more consistent comparisons than consistent scores, though regressing directly to a score performs about as well on some tasks.

The constant rater-error noise model addresses a wrinkle in the comparisons the Bradley-Terry fit depends on: because human raters are never perfectly accurate even on easy pairs, the paper adds a fixed 10% chance of a uniformly random response to the softmax, so predicted preference never asymptotes to false certainty.

The 2017 paper explains its own reward scale with an analogy to chess ratings: a difference in predicted reward should predict a preference the way a difference in Elo points predicts one player defeating another. Constitutional AI (2022) takes that analogy literally, computing an actual Elo score from crowdworker pairwise comparisons to track each model snapshot's helpfulness and harmlessness during training.

The comparison-to-scale logic this theme has been building reappears five years later, structurally unchanged, as the reward model pairwise comparison loss that trains InstructGPT's (2022) reward model: a cross-entropy loss in which the reward difference between a preferred and dispreferred completion stands for the log odds a human would choose the former. Where Bradley-Terry scored short video clips, this loss scores ranked pairs of language outputs.

All of this converges on the reward model, the artifact the comparison pipeline exists to produce: a network trained on comparisons that stands in for a missing reward function. InstructGPT (2022) trains the same canonical kind of model, initialized from its supervised fine-tuned model with a scalar output head, on human rankings of language completions rather than robot behavior clips.

The asynchronous reward-learning architecture is the design in the 2017 paper that makes every other countermeasure in this theme possible: a policy, a human-preference-collection process, and a reward predictor run as three concurrent loops, with trajectories flowing to comparisons, comparisons flowing to the predictor, and the predictor's updated parameters flowing back to the policy. Because the predictor keeps refitting while the policy keeps acting, the reward it produces is a moving target rather than a fixed function.

Iterated (online) training is the discipline that architecture enables: preference labels are collected continuously throughout training rather than only at the start. The paper's own ablation shows this matters, since training only on early labels lets the policy learn to exploit gaps in a reward predictor that never gets to see where the policy goes next.

The non-stationary reward function challenge names the cost of that same choice: because the reward predictor is repeatedly refit during training, the policy is effectively solving a reinforcement learning problem whose reward keeps changing underneath it. The paper responds by favoring policy-gradient methods that tolerate a shifting reward and by raising the entropy bonus to preserve exploration.

The reward predictor ensemble is one concrete safeguard against a predictor that overfits before it has caught up with the policy: three independently-trained predictors, each fit on a bootstrap resample of the preference data, with the final estimate given by averaging their independently-normalized outputs. Disagreement across the ensemble is also what later drives which comparisons get queried at all.

Adaptive L2 regularization for the reward predictor is the ensemble's companion safeguard: the L2 penalty coefficient on each predictor is adjusted during training to keep held-out validation loss between 1.1 and 1.5 times training loss, using a held-out fraction of 1/e of the preference database. Where the ensemble spreads risk across multiple fits, this scheme tunes each individual fit against its own overfitting.

Reward predictor pretraining addresses the earliest moment this non-stationarity could go wrong: in the Atari experiments, the reward predictor trains for 200 epochs on comparisons drawn from an untrained, randomly-initialized policy before RL begins at all. This reduces the risk that the policy converges irreversibly on a bad behavior while being guided by a predictor that has not yet learned anything useful.

Label annealing governs how the query rate changes once RL training is under way: the rate at which new preference labels are requested decreases roughly inversely with elapsed timesteps, so the predictor gets dense feedback early, when pretraining alone is not enough, and progressively sparser feedback later, once it has mostly caught up with the policy.

Reward prediction normalization is a smaller but necessary fix downstream of all this refitting: the predictor's output is rescaled to zero mean and a fixed standard deviation before the RL algorithm uses it, since a comparison-only training objective leaves the reward's absolute scale and offset undetermined.

The recurrent reward model extension closes this theme with the general case the paper's actual architecture only approximates: rather than estimating reward from a fixed window of recent observations, as the real experiments do, a recurrent network could model reward as a function of an agent's whole observation history in a partially observable environment. The paper flags this as an unimplemented extension, leaving open how much further off-distribution drift such a model would need to withstand.

The non-stationary reward function challenge is the constraint this entire theme responds to: because the reward predictor is repeatedly refit during training, a policy trained against it is solving a reinforcement learning problem whose reward keeps changing underneath it. The 2017 paper's response is to favor policy-gradient algorithms that tolerate this drift and to raise the entropy bonus to preserve exploration.

Advantage Actor-Critic (Mnih et al., 2016) is the policy-gradient algorithm the paper actually picks for its seven Atari tasks, run in its synchronous form with standard hyperparameters reused from prior work rather than retuned for the changing reward. It is the first of two domain-specific answers to the non-stationarity problem just introduced.

Trust Region Policy Optimization (Schulman et al., 2015) is the paper's second answer, chosen for the MuJoCo robotics tasks, with one hyperparameter deliberately adjusted: its entropy bonus is raised above standard settings, specifically to restore exploration that the trust region would otherwise sacrifice under a shifting reward.

Deep Q-Network (Mnih et al., 2015) appears in this theme only as a comparison point, not as a method the paper uses: the authors note that their Enduro results with real human feedback are comparable to DQN's reported performance on that game.

Reward prediction normalization is the shared preprocessing step that lets both A2C and TRPO consume the same kind of reward signal: the predictor's output is rescaled to zero mean and fixed standard deviation before either optimizer sees it, since a comparison-trained predictor leaves the reward's absolute scale otherwise undetermined.

The same non-stationarity constraint reappears five years later inside PPO fine-tuning, the RL stage of InstructGPT (2022): there, too, the policy is optimized against a reward model that keeps shifting as training proceeds, the same moving-target problem A2C and TRPO were chosen to tolerate in 2017.

Quantitative vs. qualitative evaluation methodology sets up the 2017 paper's two-track test that this whole theme is designed to make trustworthy: quantitative tasks compare a trained agent against a true reward function that exists but is withheld from training, while qualitative tasks, which have no reward function at all, are judged by a human watching video against a natural-language goal description. Making the quantitative track meaningful requires closing off every other way the true objective could leak in.

Environment modifications for preference learning do exactly that closing-off: on-screen score displays in Atari are blanked, variable-length episode termination (like resets when a MuJoCo robot falls over) is replaced with a learned penalty, and torque penalties are stripped from the Gym reward functions. Without these changes, an agent could pick up task information from the environment itself rather than from human feedback, making the quantitative comparison meaningless.

Synthetic oracle feedback is the controlled comparison this cleaned-up environment enables: a synthetic rater that always answers according to the true, withheld reward, standing in for a human so the authors can isolate the reward-learning method's own effect from the noise and cost of real raters. Comparing real human feedback against this oracle on the same modified environments is what the next finding depends on.

Human-feedback implicit reward shaping is the finding that whole apparatus surfaces: on tasks like Ant and Enduro, real human comparisons sometimes outperform the synthetic oracle drawing on the true reward, because a human's preferences shape the reward more usefully than the hand-written objective they are meant to approximate. The experimental design built to isolate the method's effect ends up revealing that human judgment can beat the ground truth it was tested against.

Hopper is a single-legged MuJoCo robot whose task is to move forward without falling over, one of the paper's standard benchmarks learned without observing the true reward. It later becomes the setting for the backflip demonstration, a behavior with no natural hand-engineered reward at all.

Half-Cheetah is a two-legged MuJoCo robot run forward as fast as possible, learned the same way as Hopper but with a different body plan. It too gets reused later, as the setting for a novel one-legged-running demonstration rather than its standard running task.

Ant is a four-legged MuJoCo robot whose task is to stay upright and move forward, and it is the theme's most notable MuJoCo result: human feedback emphasizing that the robot stand upright outperformed synthetic oracle feedback drawn from the true reward, apparently because the human comparisons shaped the reward better than the hand-crafted upright bonus did.

Atari games, accessed through the Arcade Learning Environment and the same Gym interface as MuJoCo, supply the theme's second domain: seven titles from a widely used prior RL benchmark suite, of which this theme covers three. Where MuJoCo tested continuous robot control, Atari tests discrete-action video game play under the same preference-learning method.

Enduro is an Atari racing game whose usual objective is passing cars; the paper's method, trained with real human feedback, outperforms the A3C baseline here because raters reward any progress passing cars, shaping a reward the game's usual scoring does not provide as densely. It is later modified into a pace-keeping task for one of the novel-behavior demonstrations.

Pong is a standard Atari benchmark task, notable in the ablation studies for a specific failure mode: training the reward predictor offline, without ongoing online queries, sometimes taught the agent to avoid losing points without learning to score any, producing pathologically long volleys.

Qbert closes this theme as its one outright failure: the method never learns to beat the first level using real human feedback, plausibly because Qbert's short video clips are confusing for a rater to parse and evaluate. Across all six tasks in this theme, Qbert is the clearest reminder that the method's successes were not guaranteed.

Human-feedback sample efficiency is the 2017 paper's headline number and the constraint every other concept in this theme has to satisfy: complex tasks and entirely novel behaviors get learned while a human labels less than 1% of the agent's environment interactions, roughly three orders of magnitude less than feedback on every timestep would require. Everything that follows is an accounting of how that budget gets spent.

The trajectory segment keeps each individual query cheap to begin with: rather than a whole episode, a human judges a clip of only one to two seconds of behavior, so a single query costs a few seconds of attention rather than minutes of watching. Choosing this unit is the first lever the paper pulls to make the sample-efficiency budget attainable.

Clip-length effects on human evaluation is the finding that justifies picking that particular length rather than something shorter or longer: very short clips take a rater longer to understand relative to the information they convey, so the paper measures rater evaluation time against clip duration and keeps the shortest length for which that time still scales roughly linearly.

Uncertainty-based query selection decides which of the many possible segment pairs are worth spending that budget on: a large batch of candidate pairs is sampled, every member of the reward-predictor ensemble predicts a preference for each, and the pairs where the ensemble disagrees most are the ones actually shown to a human. This spends scarce queries on the comparisons most likely to teach the predictor something.

Expected-value-of-information query selection is the more principled version of that same idea, one the paper credits to Akrour et al. (2012) and Krueger et al. (2016) but leaves to future work: choosing queries by their expected value of information rather than by how much an ensemble happens to disagree. The authors flag it as an unimplemented improvement on uncertainty-based query selection, not a component of the actual system.

The contractor preference-labeling protocol is where the human-feedback budget is actually spent: paid contractors sign up for time slots, receive a one-to-two sentence task description, and then repeatedly compare clips through a keyboard-driven interface offering 'prefer left,' 'prefer right,' 'tie,' or 'can't tell,' answering a query every few seconds. This is the concrete labor underneath every efficiency number in this theme.

Label annealing determines how that contractor labor gets allocated over the course of training: the rate at which new labels are requested decreases roughly inversely with elapsed timesteps, front-loading queries when the predictor most needs them and tapering off once it has caught up. It is the scheduling policy layered on top of the query-selection and labeling machinery already described.

Compute vs. human feedback cost analysis is the paper's own closing arithmetic on this whole budget: about $25 of Atari compute for roughly a day of training on a 16-CPU, one-GPU cloud machine turns out already comparable to about $36 of minimum-wage labor behind 5,000 labels. That rough parity is what makes the paper argue further reductions in labels needed would yield diminishing practical returns.