Human-Feedback Implicit Reward Shaping
Deep Reinforcement Learning from Human Preferences — introduced
An observed phenomenon in which reward predictors learned from human preferences sometimes outperform the ground-truth RL reward function, because human raters implicitly reward-shape the task (e.g. rewarding 'standing upright' on Ant, or rewarding any progress passing cars on Enduro), yielding denser or better-shaped guidance than the original hand-written reward.
The expectation going in was modest: comparisons might match a hand-written reward, not beat it. On two tasks, human feedback did better than the reward function it was only supposed to approximate.
The page covers where this happened and what the raters were actually rewarding, then a twist, the same advantage shows up with no human involved at all, before closing on why, once the paper turns to entirely new behaviors, this shaping stops being a bonus and becomes the whole signal.
The phenomenon
The paper's default expectation is that learning from a handful of human comparisons can approach, but not exceed, what training directly on the true reward achieves. On two tasks it does better. On the Ant robotics task, human raters instructed to prefer trajectories where the robot is "standing upright" produce a reward signal that outperforms the ground truth reward's own hand-crafted upright bonus (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7). On Enduro, human labelers "tend to reward any progress towards passing cars," implicitly shaping a dense signal out of a task whose sparse true reward is hard for A2C's asynchronous relative A3C to learn by direct exploration (deep-rl-human-prefs, §"3.1.2 Atari", p. 8).
Not uniquely a human phenomenon
The natural story is that human judgment specifically supplies better shaping than a hand-written bonus. But the paper reports the same advantage with no human in the loop at all: at 1400 labels, training against Synthetic Oracle Feedback's comparisons "performs slightly better than if it had simply been given the true reward," because "the learned reward function is slightly better shaped" (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7). Some of the shaping advantage, in other words, comes from how the reward model generalizes from comparisons, not solely from a person's judgment about what "good" looks like.
From bonus to the entire signal
On Ant and Enduro, implicit shaping is a bonus layered on top of a true reward that still exists and is used only to score the outcome, never to train on. Novel Behavior Training's three demonstrations remove that safety net entirely: there is no withheld true reward for a backflip, a one-legged gait, or driving level with traffic, so whatever shaping the raters supply during training is the entire signal the policy ever sees (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 8). A curiosity observed in §3.1 becomes, in §3.2, the thing the whole method depends on.