Novel behaviors without reward functions
Matching a benchmark's existing score proves a method works; teaching a behavior nobody could write a reward function for is the harder test, and the 2017 paper spends roughly an hour of human time per behavior finding out. A simulated robot learns to backflip and land upright, another learns to run on one leg, and a racing car learns to hold pace with traffic instead of overtaking it, none of it guided by any programmatic score.
The walk covers each demonstration in turn, closing on the finding that explains why they work: human comparisons often shape a reward more usefully than a hand-written one would.
Matching existing RL benchmarks proves the method works; teaching behaviors nobody could have hand-coded a reward for is the point of it, and novel behavior training is the paper's demonstration of exactly that, using roughly an hour of human feedback per behavior and no programmatic reward at all. The Hopper backflip demonstration trains the simulated Hopper robot, using 900 preference queries, to perform a sequence of backflips and land upright each time, a behavior with no natural hand-engineered reward. The Half-Cheetah one-leg demonstration trains the Half-Cheetah robot, using 800 queries, to move forward balanced on a single leg. The Enduro keeping-pace demonstration retrains the Atari racing task against a different goal than the game's own scoring: using roughly 1,300 queries, the agent learns to stay level with surrounding traffic rather than pass cars. Human-feedback implicit reward shaping is the reason these demonstrations work as well as they do: a human's comparisons often carry denser or better-shaped guidance than a hand-written reward would, the same shaping effect observed on the Ant and Enduro benchmark tasks.