RL from AI Feedback (RLAIF)
Constitutional AI: Harmlessness from AI Feedback — introduced
A variant of RLHF in which the harmlessness preference labels used to train the reward/preference model are generated by an AI feedback model instead of human crowdworkers.
Ask who judges which response is more harmless, and RLHF's usual answer is crowdworkers. RL from AI Feedback swaps in an AI model for that one judgment call and leaves the rest of the training pipeline untouched.
The page frames this as a single swap rather than a redesign, then as one instance of a broader idea, oversight that scales past what human labelers alone can manage, opening onto the feedback model it depends on.
Swapping the labeler, not the pipeline
RL from AI Feedback is a variant of Reinforcement learning from human feedback (RLHF) in which harmlessness preference labels come from an AI Feedback Model instead of human crowdworkers. The paper is explicit that this is a minimal, surgical change: "the remainder of the training pipeline (i.e., preference model training and RL) is exactly the same as RLHF" (constitutional-ai, §"4.1 Method", p. 10; Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)). Helpfulness comparisons remain human-labeled throughout, so every RLAIF run produces a Hybrid Human/AI Preference Model rather than a fully AI-supervised one. Mechanically, the Feedback Model is shown a harmful prompt and a pair of responses formatted as a Multiple-Choice Comparison Format question, and its answer log-probabilities become a soft preference label, later adjusted by Preference Label Clamping (Soft vs. Hard vs. Clamped) when chain-of-thought prompting is used (RL from AI Feedback (RLAIF) — delegates harmlessness comparisons to → Feedback Model).
The corpus's clearest instance of scalable oversight, realized
RLAIF is the point at which the From an open problem to reward models to AI feedback running through the whole three-paper corpus reaches its endpoint: Concrete Problems names scalable oversight as an open problem requiring cheap proxies for expensive true objectives, InstructGPT's Reward model (RM) supplies a human-labeled proxy at scale via Reinforcement learning from human feedback (RLHF), and RLAIF removes the last remaining human bottleneck for one of the two comparison axes. This is also the concept in this batch most directly relevant to a reader arriving from InstructGPT: RLAIF inherits every mechanical property of Reinforcement learning from human feedback (RLHF) as InstructGPT built it (preference-model training, PPO, the KL penalty against a reference policy) and differs only in the provenance of the harmlessness half of the comparison data. The paper also flags a natural extension the automation enables but does not itself run: Iterated (Online) Training, periodically refreshing the preference model on the current policy's outputs, is described as newly feasible "by using AI feedback we can fully automate the process" (constitutional-ai, §"6.1 Future Directions", p. 15; RL from AI Feedback (RLAIF) — makes newly automatable → Iterated (Online) Training).
Its three children
RLAIF's mechanism is built from three pieces, each a child of this concept. Feedback Model is the AI labeler itself. Multiple-Choice Comparison Format is the prompt template that turns an ordinary pretrained language model into that labeler by repurposing next-token prediction into a scalar judgment. Preference Label Clamping (Soft vs. Hard vs. Clamped) is the correction needed once chain-of-thought reasoning is added to the feedback model, since a CoT explanation that states its answer up front breaks the calibration the labeling scheme depends on. Together these three concepts are the engineering substance behind RLAIF's one-sentence description, and are gathered under the Engineering a language model into a usable preference labeler theme.