RL from AI Feedback (RLAIF) — makes newly automatable → Iterated (Online) Training

Iterated online training — periodically refreshing the preference model with new comparisons collected on the current policy's own outputs, so the PM never drifts too far off-distribution — already existed as a valuable technique using human feedback in prior work (Bai et al. 2022). Constitutional AI notes that 'we should be able to perform iterated "online" training...with AI supervision, where we update the preference model with new AI feedback in order to keep it on the same distribution as the policy produces...by using AI feedback we can fully automate the process' (constitutional-ai, §"6.1 Future Directions", p. 15). RLAIF's contribution here is not the refresh idea itself but removing its bottleneck: once an AI feedback model rather than a human is doing the labeling, the refresh loop no longer needs continuous human throughput and can in principle run indefinitely. Notably, this remains a flagged future direction, not something Constitutional AI's own experiments implement.