The 2017 deep-RL testbeds: MuJoCo and Atari
Learning from preferences needed to prove itself on tasks hard enough that the result would mean something, and the 2017 paper found them in two existing benchmark suites rather than building new ones. MuJoCo supplies simulated robots that must stay upright and move, Atari supplies arcade games with score-based objectives, and OpenAI Gym gives both a shared interface.
The walk introduces that shared interface, then MuJoCo and three of its robots, Hopper, Half-Cheetah, and Ant, before turning to Atari and three of its games, Enduro, Pong, and Qbert, where a success, a reward-hacking case, and an outright failure each play out.
The 2017 paper needed testbeds complex enough that learning from preferences would mean something, and it drew them from two existing sources accessed through a common interface, OpenAI Gym, the toolkit providing a shared API to both domains. MuJoCo supplies the physics-simulated robotics tasks, of which this theme covers three: Hopper, a single-legged robot that must move forward without falling, later also the setting for the backflip demonstration; Half-Cheetah, a two-legged robot run forward as fast as possible, later the setting for the one-legged-running demonstration; and Ant, a four-legged robot whose task is to stay upright and move forward, notable because human feedback about staying upright outperformed the synthetic oracle. Atari games (as an RL testbed) supplies the second domain, seven titles from the Arcade Learning Environment, of which this theme covers three: Enduro, a racing game whose usual objective is passing cars, later modified for the pace-keeping demonstration; Pong, notable in the ablation studies for the reward-exploiting behavior that appeared under offline reward training; and Qbert, the one task the method failed to learn from real human feedback, plausibly because its short clips are hard for a rater to parse. Across both domains the same pattern holds: the testbeds themselves are standard RL benchmarks, but the individual tasks are where the paper's successes, near-misses, and one outright failure become legible.