Hiding the reward: experimental design for preference-only learning — licenses the results of → The 2017 deep-RL testbeds: MuJoCo and Atari

The near-parity numbers the testbeds theme reports, 700 labels nearly matching TRPO on MuJoCo, 5,500 nearly matching A2C on Atari, would prove nothing about learning from preferences alone if the environments still leaked the task objective through some other channel, a dependency the testbeds theme itself does not spell out. That is exactly what the environment modifications close off: on-screen score displays are blanked, variable-length episode termination is replaced with a learned penalty, and torque penalties are stripped from the Gym reward functions, changes made specifically "to avoid encoding information about the task in the environment itself" (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 6; §"A Experimental Details", p. 14). The synthetic oracle supplies the other half of the same warrant: because it answers queries strictly from the withheld true reward, comparing real-human runs against it isolates how much of a testbed's performance came from the comparison-learning method itself rather than from residual leakage or an easy task (deep-rl-human-prefs, §"3.1 Reinforcement Learning Tasks with Unobserved Rewards", p. 6). Without this design, a benchmark score on Hopper or Pong would be uninterpretable, since nothing would rule out the agent having quietly rediscovered the true objective some other way; with it, the testbeds' numbers become a real test of preference learning rather than a demonstration that RL still works when a reward happens to be hidden somewhere nearby.