Ant (MuJoCo task)
Deep Reinforcement Learning from Human Preferences — inherited
A MuJoCo continuous-control benchmark task involving a four-legged robot that should remain upright and move forward; notable because human feedback, which emphasized keeping the robot 'standing upright,' outperformed synthetic oracle feedback, apparently providing better reward shaping than the hand-crafted upright bonus in the true reward function.
A four-legged robot's reward function can include a hand-written bonus for standing upright and still get outperformed by people simply saying what they liked. Ant is the task where that happened.
The page covers what the task asks of the robot, then the result where human feedback beat even the built-in reward's own upright bonus, and closes on the contractor instructions that made the upright priority an explicit rule rather than something raters merely picked up on their own.
What the task is
Ant is a four-legged MuJoCo robot whose task combines locomotion with balance: it should remain upright while moving forward, with reward — like the other MuJoCo tasks — a quadratic function of distances, positions and velocities (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 6). The true reward function includes "a similar bonus... to encourage the robot to remain upright" alongside its forward-progress term (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7).
Where human feedback beat the oracle
Ant is the one MuJoCo task where real human feedback clearly outperformed the Synthetic Oracle Feedback baseline: "the human feedback significantly outperformed the synthetic feedback, apparently because we asked humans to prefer trajectories where the robot was 'standing upright,' which proved to be useful reward shaping" — better shaping, the paper notes, than "the simple hand-crafted bonus" already present in the true reward (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7). This is the clearest instance in the robotics suite of Human-Feedback Implicit Reward Shaping: raters improved on a term the authors themselves had already hand-engineered.
An explicit instruction, not an implicit preference
The upright priority is often summarized as something humans convey implicitly, but the Contractor Preference-Labeling Protocol edge shows the contractor instructions stated it outright: "the first priority is for the robot to be standing upright, and failing that for the center of the robot to be as high up as possible," with forward speed demoted to a tiebreaker, "whichever one is moving faster to the right" (deep-rl-human-prefs, §"B.1 MuJoCo", p. 16). The reward-shaping win traces to this explicit ranking of priorities, not to some tacit human sense the reward model happened to pick up.