Novel Behavior Training
Deep Reinforcement Learning from Human Preferences — introduced
A set of experiments demonstrating that the method can teach agents entirely new behaviors that were never associated with a programmatic reward function, specified only via roughly an hour of human feedback: a robot performing repeated backflips, a half-cheetah moving forward on one leg, and a car agent that drives level with surrounding traffic.
Matching a benchmark that already has a reward function is one kind of test. Teaching a behavior for which no reward function has ever existed, or could easily be written, is a much harder one.
The page introduces the three demonstrations gathered here, then the reward model's deliberately oversized capacity built in anticipation of exactly this moment, and closes on why, with no hidden true reward left to fall back on, these three results are the paper's clearest proof of its own stated goal.
Three demonstrations beyond benchmark-matching
The paper's first set of experiments (§3.1) shows the method can match standard RL benchmarks even when the true reward is hidden from the agent. Novel Behavior Training is the more ambitious second act: three tasks for which no natural reward function exists at all. The Hopper Backflip Demonstration trains the simulated Hopper robot to perform a sequence of backflips and land upright each time; the Half-Cheetah One-Leg Demonstration trains Half-Cheetah to move forward while balancing on a single leg; and the Enduro Keeping-Pace Demonstration trains an Enduro-playing agent to drive level with traffic instead of racing past it (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 8). Each behavior reuses the same algorithm, network architectures, and hyperparameters as the benchmark experiments, needing only 800 to roughly 1,300 preference queries and under an hour of feedback, this time supplied directly by the paper's own authors rather than outsourced contractors.
A reward model over-built for this moment
The two-layer neural network used to predict reward on the MuJoCo tasks is more expressive than any of the standard benchmarks strictly require: the paper notes their true reward functions, Hopper and Half-Cheetah included, are all just second-degree polynomials of position, velocity, and distance features, simple enough that a much simpler regression method would have sufficed (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14). That surplus capacity is deliberate: the same footnote explains it exists so the Reward model (RM) can generalize immediately to tasks like a backflip or a one-legged gait, which have no such clean functional form, without any change of architecture (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14).
Where implicit shaping stops being a bonus
On Ant, Human-Feedback Implicit Reward Shaping is a pleasant surprise layered on top of a real reward: human comparisons happen to outperform a true, withheld reward that still exists and is used only to score the outcome (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7). In Novel Behavior Training there is no such fallback. No true reward for a backflip, a one-legged gait, or driving level with traffic exists to compare against, so whatever shaping the raters supply during training is the entire signal the policy ever optimizes (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 8). That is what makes these three demonstrations the paper's proof of its stated goal: solving "tasks for which we can only recognize the desired behavior, but not necessarily demonstrate it" (deep-rl-human-prefs, §"1 Introduction", p. 2).