Label Annealing
Deep Reinforcement Learning from Human Preferences — introduced
A schedule that decreases the rate at which new preference labels are requested over the course of training, roughly inversely proportional to elapsed timesteps, balancing the value of a good reward predictor early in training against the need to keep adapting it as the policy visits new states.
Asking for a comparison too rarely risks a reward model that never catches up with what the policy is doing; asking too often burns through a scarce and expensive resource. This is the schedule built to balance the two.
The page describes the decaying rate at which labels are requested over training, then the two competing needs it balances, and closes on why the schedule is only ever approximate once real contractors, rather than a scripted process, are the ones answering.
A decaying request rate
Label annealing decreases the rate at which new preference queries are answered over training, roughly in inverse proportion to elapsed timesteps: after twice as many timesteps have passed, the system asks for about half as many labels per unit time (deep-rl-human-prefs, §"A Experimental Details", p. 14). In the MuJoCo experiments, 25% of comparisons come from a randomly initialized policy at the start of training, and the labeling rate after $T$ frames follows $2\times10^6/(T+2\times10^6)$ (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14); the Atari rate follows the analogous $5\times10^6/(T+5\times10^6)$, decremented every $5\times10^6$ frames (deep-rl-human-prefs, §"A.2 Atari", p. 15).
Balancing an early need against a moving target
The schedule balances two needs the paper names directly: having "a good predictor from the start" against "the need to adapt the predictor as the RL agent learns and encounters new states" (deep-rl-human-prefs, §"A Experimental Details", p. 14). It supplies the pacing that iterated online training itself leaves unspecified — that concept only establishes that labels keep arriving throughout training rather than being front-loaded, not how quickly they should arrive. That pacing decision is why the concept sits in the The economics of human feedback theme alongside its role in keeping the reward model on-distribution.
Only approximate with real contractors
The schedule is followed exactly when a synthetic oracle supplies labels, since the experiment can request a comparison on whatever cadence it likes. Real contractors break that precision: the paper states that with human feedback "we attempt to similarly anneal the label rate, although in practice this is approximate because contractors give feedback at uneven rates" (deep-rl-human-prefs, §"A Experimental Details", p. 14). The Hopper results show visibly irregular progress traced to one contractor deviating from the intended labeling schedule (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7).