SL-CAI (Supervised-Learning Constitutional AI model) — seeds the policy and comparison data for → RL-CAI (RLAIF-trained Constitutional AI model)

sl-cai plays two roles in rl-cai’s training at once: it is the initial snapshot the RL run starts from, and it generates the response pairs the feedback model labels. The paper is explicit that this doubling is deliberate: "We suspect that using the same model for both should lead to better results, since the distribution of responses generated by the policy are similar to the preference model training distribution, at least during early phases of RL" (constitutional-ai, §"4.1 Method", p. 11). Starting RL from helpful-rlhf-model directly would forfeit both properties — the policy would begin off-distribution relative to the AI-labeled comparisons, and the preference model would be scoring responses unlike anything the policy produces. The supervised stage exists largely for this handoff: its stated purpose is to get the model "on-distribution" and "reduce the need for exploration and the total length of training during the second RL phase" (§"1.2 The Constitutional AI Approach", p. 5), addressing the exploration problems flagged in the Figure 1 caption (p. 2).