Half-Cheetah (MuJoCo task)
Deep Reinforcement Learning from Human Preferences — inherited
A MuJoCo continuous-control benchmark task requiring a two-legged 'cheetah' robot to run forward as fast as possible; used as a standard RL benchmark and as the setting for the one-legged-running novel behavior.
Running forward as fast as possible is an easy thing to reward by hand, until the same robot is asked to do something no formula was ever written for.
The page describes the standard task and its reward, then a quirk in how its main result was actually collected, before turning to how little extra it cost to teach the same robot an entirely new gait instead.
What the task is
Half-Cheetah is a two-legged MuJoCo robot whose standard objective is to run forward as fast as possible; contractors were simply told that "the robot should move to the right as fast as possible" (deep-rl-human-prefs, §"B.1 MuJoCo", p. 16). Like the other MuJoCo tasks, its reward is "a quadratic function" of distances, positions, and velocities (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 6), and it received the same termination and torque-penalty modifications applied across the suite.
An exception to the contractor pipeline
Half-Cheetah is one of only two MuJoCo tasks, with Reacher, whose Figure 2 result did not come from paid contractors: "for Reacher and Cheetah feedback was provided by an author due to time constraints" (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7), per the Contractor Preference-Labeling Protocol edge — even though the paper's general practice elsewhere used contractors "unfamiliar with the environments and with our algorithm."
Barely more expensive to teach a new behavior
The standard benchmark, run against Half-Cheetah's existing hand-engineered reward, was matched with 700 preference queries (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 6). Teaching the same robot an entirely new behavior with no reward function to hand-engineer — standing and moving forward on one leg — took only 800 queries, "trained... in under an hour" (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 9), via the Half-Cheetah One-Leg Demonstration. The gap between matching an existing objective and inventing a new one is, in query terms, small.