Hiding the reward: experimental design for preference-only learning
Showing an agent learned a behavior from preferences alone, not some leaked score or timer, took almost as much design work as the learning method itself. The 2017 paper strips its environments of every side channel, on-screen scores, variable episode length, hand-written torque penalties, so nothing but a rater's judgment carries information about the task.
The walk moves from the two-track evaluation scheme, quantitative tasks against a withheld true reward and qualitative tasks judged by a human, through the environment scrubbing that makes it fair, to a synthetic oracle standing in for a perfect rater, closing on the finding that real humans sometimes beat it.
Proving that an agent learned a behavior from preferences alone, and not from some other leaked signal, took as much experimental design as the learning method itself. The quantitative vs. qualitative evaluation methodology sets up the two-track test: quantitative tasks compare the trained agent against a true reward function withheld from training, while qualitative tasks, where no reward function exists at all, are judged by a human watching video against a natural-language goal description. Environment modifications for preference learning close off the side channels that would make quantitative comparisons meaningless: on-screen score displays are blanked, variable-length episode termination is replaced with a learned penalty, and torque penalties are stripped from the Gym reward functions, so nothing beyond the human's feedback carries information about the task. Synthetic oracle feedback is the controlled comparison this design enables: a synthetic rater that always answers according to the true, withheld reward, letting the authors isolate the reward-learning method's own effect from the noise and cost of real human raters. Human-feedback implicit reward shaping is the finding this whole apparatus surfaces: on tasks like Ant and Enduro, real human comparisons sometimes outperform the synthetic oracle, because a human's preferences shape the reward more usefully than the hand-written objective they are approximating.