MuJoCo
Deep Reinforcement Learning from Human Preferences — inherited
A physics simulator (Todorov et al., 2012) used to implement the paper's simulated-robotics locomotion and control tasks (Hopper, Walker, Swimmer, Half-Cheetah, Ant, Reacher, Pendulum, Double-pendulum), accessed via OpenAI Gym.
Eight simulated robots, from a simple pendulum up to a four-legged Ant, all share the same physics engine and the same style of hand-written reward underneath them.
The page introduces the simulator and how the paper reaches it through a common interface, then the changes made to its default termination and reward rules so a hidden true reward wouldn't leak through some other channel, and closes on how well the method actually did against that hidden reward.
What it is and how it's used here
MuJoCo is the physics simulator (Todorov et al., 2012) underlying all eight of the paper's simulated-robotics tasks, accessed through OpenAI Gym rather than called directly (deep-rl-human-prefs, §"3 Experimental Results", p. 6). The tasks span a simple "pendulum," included "since this is representative of the complexity of tasks studied in prior work," up through Hopper (MuJoCo task), Half-Cheetah (MuJoCo task), and Ant (MuJoCo task) (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 6). All eight share a reward structure: "linear functions of distances, positions and velocities... all are a quadratic function of the features" (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 6).
Modifications made for preference learning
Standard Gym/MuJoCo episodes end early when the robot's state leaves a prescribed range, e.g. falling over — a termination signal the paper found leaks task information even without an observable reward. It was replaced by "a penalty which encourages the parameters to remain in the range (and which the agent must learn)" (deep-rl-human-prefs, §"A Experimental Details", p. 14), per the Environment Modifications for Preference Learning concept. Gym's built-in torque penalties were also stripped out, since "torques are not directly visible to a human supervisor" (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14).
Its results in the paper
With 700 preference queries, the method is "able to nearly match reinforcement learning on all of these tasks" (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7), and by 1400 synthetic labels it sometimes exceeds RL on the true reward, attributed to a better-shaped learned reward. Feedback came from contractors on most tasks, but from an author on Reacher and Half-Cheetah "due to time constraints" (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7).