Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)

RLAIF is presented as a minimal, surgical modification of RLHF rather than a new algorithm: Constitutional AI states that after the AI feedback model produces the comparison labels, 'the remainder of the training pipeline (i.e., preference model training and RL) is exactly the same as RLHF' (constitutional-ai, §"4.1 Method", p. 10). The only thing that changes is who answers the comparison question for harmlessness — an independent feedback model instead of a crowdworker — while helpfulness comparisons remain human-labeled throughout, so every RLAIF run is quietly a human/AI hybrid rather than a fully AI-supervised system. This framing matters for how to read RLAIF: it inherits every mechanical property of RLHF (preference-model training, PPO, the KL penalty against the SFT policy) and differs only in the provenance of one input dataset.