Deep RL from Human Preferences: the 2017 proof of concept
Turning human preferences into a usable training signal for deep RL took several ideas at once: a cheap way to elicit them, engineering against an exploited reward, and algorithms built for a moving target. The 2017 paper is the full proof of concept, and its own cost accounting, an hour of human time per behavior, made it look practical rather than merely possible.
The walk opens on the prior methods it outgrew, moves through the comparison mechanism, the safeguards, and the algorithms built to survive a shifting reward, and closes on the testbeds and novel behaviors, backflip included, that proved the method actually worked.
This supertheme is the 2017 paper's full proof of concept: everything it took to make human preferences into a usable training signal for deep reinforcement learning, from the mechanism to the countermeasures to the proof that it worked. Prior human-in-the-loop RL and why it didn't scale sets up the gap the paper closes, a decade of demonstration-based, evaluative-feedback, and cooperative-game approaches each blocked from deep RL by needing expert demonstrations, hand-coded features, or constant human attention. Comparisons as the atomic unit of human feedback is the mechanism that closes it: humans compare pairs of trajectory segments rather than score them, and Bradley-Terry machinery turns those comparisons into a scalar reward. Keeping the reward model on-distribution covers the engineering that keeps that mechanism from breaking down once a policy starts optimizing against it, online labels, ensembles, regularization, and normalization, all needed because a static learned reward is exploitable. Policy optimization under a learned, changing reward is the RL side of the same problem, the reason A2C and TRPO were chosen specifically because the reward they optimize keeps moving under them. The economics of human feedback is the constraint that makes the whole design defensible: every choice traces back to a budget of minutes of non-expert attention, and the paper's own cost accounting is what made oversight of deep RL look practical for the first time. The 2017 deep-RL testbeds: MuJoCo and Atari is where the method first had to scale, and hiding the reward: experimental design for preference-only learning is how the authors proved it was really learning from preferences alone, stripping every side channel and building a synthetic oracle as a controlled comparison. Novel behaviors without reward functions is the payoff all of this was for: a backflip, a one-legged run, and even-paced driving, tasks nobody could specify a reward for, trained in under an hour of human time each.