OpenAI Gym
Deep Reinforcement Learning from Human Preferences — inherited
A toolkit (Brockman et al., 2016) providing a common interface to the MuJoCo and Arcade Learning Environment tasks used in the paper's experiments.
Two very different testbeds, a physics simulator and an arcade emulator, needed one common interface before the paper could run experiments on either the same way.
The page covers what the toolkit provides, then the specific places the paper edits its default reward and termination logic, and closes on its role as shared plumbing underneath the two very different task suites the paper actually varies.
What it is
OpenAI Gym is the toolkit (Brockman et al., 2016) the paper uses as the common interface to both testbeds: "we interface with MuJoCo... and the Arcade Learning Environment... through the OpenAI Gym" (deep-rl-human-prefs, §"3 Experimental Results", p. 6). It supplies the standard task implementations for the MuJoCo locomotion tasks such as Hopper (MuJoCo task), Half-Cheetah (MuJoCo task), and Ant (MuJoCo task), as well as access to the Atari games run through the Arcade Learning Environment.
Where the paper edits Gym's defaults
Two of the paper's environment modifications change Gym's own built-in reward and termination logic specifically. Gym's continuous-control tasks "penalize large torques," but since "torques are not directly visible to a human supervisor, these reward functions are not good representatives of human preferences over trajectories," the paper removes them (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14). Separately, in the Gym versions of the robotics tasks "the episode ends when certain parameters go outside of a prescribed range (for example when the robot falls over)"; this termination condition was replaced with a learned penalty rather than a hard reset, per the Environment Modifications for Preference Learning concept (deep-rl-human-prefs, §"A Experimental Details", p. 14).
Its place among the testbeds
Gym is shared plumbing rather than a testbed itself — the paper's experimental variety comes from the tasks it exposes, split into the MuJoCo robotics suite and seven Atari games. Both halves of the paper's quantitative evaluation (§3.1.1 and §3.1.2) depend on Gym supplying a stable, standard task interface, so that the paper's reward-learning results can be compared directly against ordinary RL trained on the environment's original reward function.