Label Annealing — paces the query rate within → Iterated (Online) Training

Iterated online training says only that labels keep arriving throughout policy training rather than all at once up front; it does not by itself specify how fast they should arrive. Label annealing supplies that schedule: the rate at which new queries are answered decays roughly inversely with elapsed timesteps, so that after twice as many timesteps have passed the system asks for about half as many labels per unit time (deep-rl-human-prefs, §"A Experimental Details", p. 14). This is a deliberate compromise rather than a default: front-loading feedback gives the reward predictor a usable signal early, while the continuing trickle of later labels is what keeps it from going stale as the policy visits states the earlier labels never covered. Even when the labels come from real contractors rather than a synthetic oracle, the paper still tries to approximate this same decaying schedule, though only approximately, since contractors do not give feedback at a constant rate.