Hopper Backflip Demonstration
Deep Reinforcement Learning from Human Preferences — introduced
A novel behavior in which the simulated Hopper robot is trained, using 900 human preference queries in under an hour, to perform a sequence of backflips and land upright each time, despite there being no natural hand-engineered reward function for 'doing a backflip.'
Nobody has written a reward function that scores a backflip. A person watching a clip can tell instantly whether one landed cleanly, which is exactly the gap this demonstration is built to cross.
The page covers what the agent was trained to do and with how little feedback, then why the ordinary reward function for this robot could never have captured a backflip in the first place, and closes on why this particular result became one of the paper's most memorable.
What the agent learned
Using the identical Hopper robot and action space as the standard MuJoCo benchmark, the paper trains the agent, via 900 pairwise preference queries collected in under an hour, to perform a repeating sequence of backflips and land upright each time (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 8; Figure 4). Feedback was supplied by the paper's own authors rather than by outsourced contractors, and the same reward-model architecture and RL hyperparameters used for Hopper's ordinary forward-locomotion benchmark were reused unchanged (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 8).
No reward function to fall back on
The ordinary Hopper task already has an easy reward to hand-engineer: the paper describes all of the MuJoCo benchmarks' reward functions, Hopper included, as simple quadratic functions of the robot's distances, positions, and velocities (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 6). "Doing a backflip" has no such natural functional form: there is no obvious combination of joint angles and velocities that scores a flip as opposed to a stumble. The demonstration extends past what Hopper (MuJoCo task)'s reward could ever specify, relying entirely on the reward model's learned fit to human comparisons rather than on any hand-written function (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 8).
Why it became a landmark result
The backflip uses the same Novel Behavior Training pipeline as the half-cheetah and Enduro demonstrations, so within the paper it is illustrative rather than exceptional. What makes it durable as an example is that a human evaluator can recognize a good backflip instantly on sight while having no idea how to write a reward function for one, exactly the gap between recognizing and specifying a behavior that the paper's introduction names as its motivating problem (deep-rl-human-prefs, §"1 Introduction", p. 1).