Iterated (Online) Training
Constitutional AI: Harmlessness from AI Feedback — inherited; Deep Reinforcement Learning from Human Preferences — introduced
Continually refitting a learned reward or preference model on comparisons drawn from the policy's own current outputs, rather than fitting it once on an early fixed batch, so the model does not drift off the distribution the policy is actually producing. Introduced in 2017 as one of three concurrent training loops, and named and flagged for full automation with AI-generated feedback in Constitutional AI's 2022 RLAIF pipeline.
A model fit once on an early batch of comparisons and then left alone can drift out of step with a policy that keeps changing underneath it. Iterated online training is the fix: keep refitting the model throughout, on whatever the policy is doing right now.
The page traces the technique to its 2017 introduction as one of three concurrent loops in the paper's own architecture, covers the ablation showing what happens when the refresh is switched off, and closes on Constitutional AI (2022) naming the same discipline as a future direction that AI-generated feedback could finally make fully automatic.
Introduced in 2017 as one of three concurrent loops
Iterated (online) training is not merely something Constitutional AI cites secondhand as "a training regime from prior work"; the 2017 paper is that prior work, and this is exactly the refresh discipline built into its Asynchronous Reward-Learning Architecture. The reward predictor is one of three processes running concurrently for the entire duration of training: the policy produces trajectories, pairs of segments are sent to a human for comparison, and the predictor is refit by supervised learning to the comparisons collected so far, with parameters for the predictor flowing continuously back to the policy rather than being fixed after a single offline fitting pass (deep-rl-human-prefs, §"2.2 Our Method", p. 4). Concretely, new labels arrive throughout training at a rate that decays roughly inversely with elapsed timesteps, governed by Label Annealing, rather than being gathered once upfront (Label Annealing — paces the query rate within → Iterated (Online) Training; deep-rl-human-prefs, §"A Experimental Details", p. 14).
Why it matters: the 'no online queries' ablation
The paper does not just assert that online refresh is important, it demonstrates it by removing it. The "no online queries" ablation trains the reward predictor only on comparisons gathered at the very start of training, then freezes it for the remainder of the run (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 9). This static predictor is what produces the paper's own worked example of Reward hacking (avoiding): on Pong, optimizing against a frozen, only-partially-correct reward leads the agent to avoid losing points without learning to score, producing extremely long volleys that repeat the same sequence of events, a behavior the paper explicitly traces to the predictor capturing "only part of the true reward" due to the nonstationarity of the occupancy distribution, citing Amodei et al. 2016 directly (Iterated (Online) Training — guards against → Reward hacking (avoiding); deep-rl-human-prefs, §"3.3 Ablation Studies", p. 9). The paper draws the general lesson explicitly: "this type of behavior demonstrates that in general human feedback needs to be intertwined with RL learning rather than provided statically" (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 10). This is also the direct cause of the Non-Stationary Reward Function Challenge: because comparisons keep arriving and the predictor keeps being refit throughout training, the policy is, at every point, optimizing against a target that is itself still moving (Iterated (Online) Training — is the cause of → Non-Stationary Reward Function Challenge).
Constitutional AI's inheritance, and automating it with AI feedback
Constitutional AI inherits this discipline directly, attributing it in name to Bai et al. (2022), and does not implement it in its own experiments; instead it names iterated online training as a future direction made newly practical by RL from AI Feedback: "we should be able to perform iterated 'online' training...with AI supervision, where we update the preference model with new AI feedback in order to keep it on the same distribution as the policy produces...by using AI feedback we can fully automate the process" (constitutional-ai, §"6.1 Future Directions", p. 15). RLAIF's contribution here is not the refresh idea itself, introduced by the 2017 paper as a structural necessity of its architecture, but removing its bottleneck: once an AI feedback model rather than a human crowdworker supplies the comparisons, the refresh loop no longer needs continuous human labor and can in principle run indefinitely (RL from AI Feedback (RLAIF) — makes newly automatable → Iterated (Online) Training). This sits at the far end of the From an open problem to reward models to AI feedback theme's throughline from a named open problem to a scaled solution, and inside the Keeping the reward model on-distribution theme it is one more instance of the Replacing the human in the loop at exactly the point it was the bottleneck connective theme's recurring move: an AI process substituting for a human one at exactly the point human throughput was the constraint.