Environment Modifications for Preference Learning
Deep Reinforcement Learning from Human Preferences — introduced
Modifications made to standard Gym/Atari environments to prevent the environment itself from leaking task-objective information outside of human feedback: removing variable-length episode termination (replacing 'falling over' resets with a learned penalty in MuJoCo, disabling life-loss/episode-end signals in Atari), removing torque penalties from Gym reward functions, and blanking on-screen score displays in Atari.
An experiment claiming an agent learned entirely from human comparisons only proves what it claims if nothing else is quietly telling the agent what the goal is. Ordinary game and simulator defaults turned out to leak exactly that kind of information.
The page explains why the environments needed editing at all, then what specifically changed, termination conditions, torque penalties, on-screen score displays, before closing on the one exemption, a single game that kept its natural ending under one particular feedback source.
Why the environment itself had to be edited
The paper's premise, that an agent can learn a complex task purely from comparisons, only tests what it claims to test if the environment isn't quietly leaking the objective through some other channel. The authors found that ordinary termination conditions do exactly this: "many RL environments have termination conditions that depend on the behavior of the agent, such as ending an episode when the agent dies or falls over... such termination conditions encode information about the task even when the reward function is not observable" (deep-rl-human-prefs, §"A Experimental Details", p. 14). Removing that leakage was necessary so that "human feedback provides its only guidance about what it ought to do" (deep-rl-human-prefs, §"A Experimental Details", p. 14) — the modifications exist to make the paper's hidden-reward experimental design airtight, not merely to tidy up the environments.
What was actually changed
In MuJoCo tasks accessed through OpenAI Gym, episodes previously ended when parameters left a prescribed range (e.g., the robot falling over); this was replaced with "a penalty which encourages the parameters to remain in the range (and which the agent must learn)" (deep-rl-human-prefs, §"A Experimental Details", p. 14). Gym's built-in torque penalties were also removed, "because torques are not directly visible to a human supervisor" and so "these reward functions are not good representatives of human preferences over trajectories" (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14). In Atari, "we do not send life loss or episode end signals to the agent... effectively converting the environment into a single continuous episode" (deep-rl-human-prefs, §"A Experimental Details", p. 14), and the on-screen score display is replaced with "a constant black background on all seven games," with BeamRider's enemy-ship counter and Enduro's speedometer additionally blanked (deep-rl-human-prefs, §"A.2 Atari", p. 15).
One exemption: Pong under the oracle
The rule is not applied with total uniformity. Under Synthetic Oracle Feedback specifically, "we replace episode ends with a penalty in all games except Pong; the agent must learn this penalty" (deep-rl-human-prefs, §"A Experimental Details", p. 14) — Pong (Atari game) alone keeps its natural episode boundary when the feedback source is the oracle rather than a human or the true reward directly. The paper does not explain the exemption further, but it shows that even this deliberately "clean" experimental setup has task-by-task carve-outs rather than one modification schema applied everywhere without exception.