Half-Cheetah One-Leg Demonstration
Deep Reinforcement Learning from Human Preferences — introduced
A novel behavior in which the simulated Half-Cheetah robot is trained, using 800 human preference queries in under an hour, to move forward while balancing and standing on a single leg.
Teaching a robot to run is one thing; teaching it to run while balancing on a single leg, a behavior nobody has written a reward function for, is another. Here the second one cost barely more than the first.
The page covers what the trained agent actually does, then how close its query budget came to the ordinary benchmark's, and closes on why a behavior like this resists being scored by any hand-written function in the first place.
What the agent learned
The paper trains the simulated Half-Cheetah robot, via 800 preference queries gathered in under an hour, to move forward while balancing and standing on a single leg (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 9). It reuses the identical robot model, action space, and reward-model architecture as the standard Half-Cheetah benchmark, with feedback again supplied by the paper's own authors rather than contractors (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 9).
Barely any more expensive than the ordinary task
Run against its own existing hand-engineered reward, the standard Half-Cheetah benchmark was matched using 700 human preference queries (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 6). Teaching the same robot an entirely new gait, one for which no reward function exists to hand-engineer at all, cost only 100 queries more: 800 total, still in under an hour (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 9). The near-equal price supports the paper's efficiency claim directly: specifying a genuinely novel goal through comparisons is not markedly harder than matching a goal someone already wrote down as a reward function.
Why one-legged running has no natural reward
A hand-written reward for Half-Cheetah (MuJoCo task) can score forward velocity directly from the simulator's state. Scoring "balance on one leg specifically" instead requires deciding in code which leg, how much ground contact from the others still counts as touching, and how to penalize the robot for reverting to two legs, decisions a human rater makes instantly by watching a clip but that resist being written as a closed-form function of joint torques and positions. This is the same recognize-but-cannot-specify gap that Novel Behavior Training as a whole is built to demonstrate.