The economics of human feedback

Members — a guided walk

1. Human-Feedback Sample Efficiency

   Human-feedback sample efficiency is the 2017 paper's headline number and the constraint every other concept in this theme has to satisfy: complex tasks and entirely novel behaviors get learned while a human labels less than 1% of the agent's environment interactions, roughly three orders of magnitude less than feedback on every timestep would require. Everything that follows is an accounting of how that budget gets spent.

2. Trajectory Segment

   The trajectory segment keeps each individual query cheap to begin with: rather than a whole episode, a human judges a clip of only one to two seconds of behavior, so a single query costs a few seconds of attention rather than minutes of watching. Choosing this unit is the first lever the paper pulls to make the sample-efficiency budget attainable.

3. Clip-Length Effects on Human Evaluation

   Clip-length effects on human evaluation is the finding that justifies picking that particular length rather than something shorter or longer: very short clips take a rater longer to understand relative to the information they convey, so the paper measures rater evaluation time against clip duration and keeps the shortest length for which that time still scales roughly linearly.

4. Uncertainty-Based Query Selection (Ensemble Variance)

   Uncertainty-based query selection decides which of the many possible segment pairs are worth spending that budget on: a large batch of candidate pairs is sampled, every member of the reward-predictor ensemble predicts a preference for each, and the pairs where the ensemble disagrees most are the ones actually shown to a human. This spends scarce queries on the comparisons most likely to teach the predictor something.

5. Expected-Value-of-Information Query Selection

   Expected-value-of-information query selection is the more principled version of that same idea, one the paper credits to Akrour et al. (2012) and Krueger et al. (2016) but leaves to future work: choosing queries by their expected value of information rather than by how much an ensemble happens to disagree. The authors flag it as an unimplemented improvement on uncertainty-based query selection, not a component of the actual system.

6. Contractor Preference-Labeling Protocol

   The contractor preference-labeling protocol is where the human-feedback budget is actually spent: paid contractors sign up for time slots, receive a one-to-two sentence task description, and then repeatedly compare clips through a keyboard-driven interface offering 'prefer left,' 'prefer right,' 'tie,' or 'can't tell,' answering a query every few seconds. This is the concrete labor underneath every efficiency number in this theme.

7. Label Annealing

   Label annealing determines how that contractor labor gets allocated over the course of training: the rate at which new labels are requested decreases roughly inversely with elapsed timesteps, front-loading queries when the predictor most needs them and tapering off once it has caught up. It is the scheduling policy layered on top of the query-selection and labeling machinery already described.

8. Compute vs. Human Feedback Cost Analysis

   Compute vs. human feedback cost analysis is the paper's own closing arithmetic on this whole budget: about $25 of Atari compute for roughly a day of training on a 16-CPU, one-GPU cloud machine turns out already comparable to about $36 of minimum-wage labor behind 5,000 labels. That rough parity is what makes the paper argue further reductions in labels needed would yield diminishing practical returns.

Superthemes

• Deep RL from Human Preferences: the 2017 proof of concept
• Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline

Edges explored within this theme

• Clip-Length Effects on Human Evaluation — calibrates the length of → Trajectory Segment
• Uncertainty-Based Query Selection (Ensemble Variance) — is a crude approximation of → Expected-Value-of-Information Query Selection
• Contractor Preference-Labeling Protocol — grounds the headline efficiency claim of → Human-Feedback Sample Efficiency
• Compute vs. Human Feedback Cost Analysis — caps the marginal value of → Human-Feedback Sample Efficiency
• Clip-Length Effects on Human Evaluation — predicts but was never tested on → Qbert (Atari game)
• Label Annealing — is only approximated under → Contractor Preference-Labeling Protocol

Super edges touching this theme

• The economics of human feedback — underwrites choosing → Comparisons as the atomic unit of human feedback
• Prior human-in-the-loop RL and why it didn't scale — fails the budget test set by → The economics of human feedback
• The economics of human feedback — supplies the cost argument behind → Building the human side of the RLHF pipeline hindsight
• The economics of human feedback — prices the implicit budget of → Scaling oversight by predicting reward on unlabeled episodes hindsight
• Three levers on one model: prompting, finetuning, steering — re runs the cost argument of → The economics of human feedback hindsight

Sources

Every passage behind this theme’s member concepts.