Hopper (MuJoCo task)
Deep Reinforcement Learning from Human Preferences — inherited
A MuJoCo continuous-control benchmark task in which a single-legged robot must hop/move forward without falling; used both as a standard RL benchmark learned from unobserved reward and as the setting for the backflip novel-behavior demonstration.
A single-legged robot that has to hop forward without falling sounds like a small thing to reward, and mostly it is, except for one run whose progress looks stranger than the rest.
The page covers the task and its reward, then the irregular result traced to a labeling schedule rather than the algorithm, and closes on how the same robot was reused for a behavior, repeated backflips, that has no reward function to write down at all.
What the task is
Hopper is a single-legged MuJoCo robot that must move forward without falling over, one of eight MuJoCo locomotion tasks accessed through OpenAI Gym (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 6). Its reward, like the other MuJoCo tasks, is "a quadratic function" of the robot's distances, positions, and velocities (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 6). Contractors were instructed that "the first priority is for the center of the robot to move to the right (moving to the left is worse than not moving at all)," with height as a tiebreaker (deep-rl-human-prefs, §"B.1 MuJoCo", p. 15).
What happened in the paper's experiments
With 700 human preference queries, Hopper is "able to nearly match reinforcement learning" trained on the true reward (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7), but its Figure 2 learning curve is visibly more irregular than the other tasks' — a wrinkle the paper traces to labeling logistics, not the algorithm: "the irregular progress on Hopper is due to one contractor deviating from the typical labeling schedule" (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7), per the Contractor Preference-Labeling Protocol edge.
Reused for a behavior with no reward function
Hopper's standard reward is easy to hand-engineer, but the Hopper Backflip Demonstration retrains the identical robot and action space toward a behavior — repeated backflips, landing upright — with no reward function written down at all, using only qualitative human judgment. It was trained "using 900 queries in less than an hour," and "the agent learns to consistently perform a backflip, land upright, and repeat" (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 8).