Human-Feedback Sample Efficiency
Deep Reinforcement Learning from Human Preferences — introduced
The paper's headline efficiency result: complex RL tasks and novel behaviors can be learned while a human provides feedback on less than 1% of the agent's environment interactions, reducing the interaction complexity of learning from human oversight by roughly three orders of magnitude compared to per-timestep feedback.
Supervising every single timestep of an agent's experience would make human attention the bottleneck on everything else. The paper's headline result is how little of that attention turned out to be necessary.
The page states the claim itself, then converts it into an actual labor cost in contractor hours, before closing on the ceiling that limits how much further squeezing the label count down would even matter.
The headline claim
The paper's central empirical result is that its method needs vastly less human attention than naive per-timestep supervision would require. The abstract states the method can "effectively solve complex RL tasks without access to the reward function... while providing feedback on less than 1% of our agent's interactions with the environment" (deep-rl-human-prefs, §"Abstract", p. 1). Section 4 restates this as an efficiency multiplier: learning a separate reward model from comparisons "reduce[s] the interaction complexity by roughly 3 orders of magnitude" relative to using human feedback directly as a per-step reward signal (deep-rl-human-prefs, §"4 Discussion and Conclusions", p. 10). This answers the problem stated in the introduction — that "using human feedback directly as a reward function is prohibitively expensive for RL systems that require hundreds or thousands of hours of experience" (deep-rl-human-prefs, §"1 Introduction", p. 2).
Converting the claim into human time
A percentage of interactions is not, by itself, a labor cost; the Contractor Preference-Labeling Protocol edge supplies the conversion. Contractors "responded to the average query in 3-5 seconds, and so the experiments involving real human feedback required between 30 minutes and 5 hours of human time" (deep-rl-human-prefs, §"3.1 Reinforcement Learning Tasks with Unobserved Rewards", p. 6). The abstract's other showcase number, that "we can successfully train complex novel behaviors with about an hour of human time" (deep-rl-human-prefs, §"Abstract", p. 1), rests on this same conversion applied to demonstrations such as the Hopper Backflip Demonstration.
A ceiling on how much the efficiency matters
The Compute vs. Human Feedback Cost Analysis edge caps the practical significance of squeezing the label count further: immediately after reporting the roughly three-orders-of-magnitude reduction, the discussion adds that the method is "already hitting diminishing returns on further sample-complexity improvements" (deep-rl-human-prefs, §"4 Discussion and Conclusions", p. 10), because, as the next page details, the cost of compute has already caught up to the cost of non-expert human feedback (deep-rl-human-prefs, §"4 Discussion and Conclusions", p. 11). Within the corpus, this result is the paper's concrete instantiation of the broader scalable-oversight problem: demonstrating empirically, in deep RL specifically, that supervision can scale sublinearly with the number of agent interactions.