SL-CAI (Supervised-Learning Constitutional AI model)
Constitutional AI: Harmlessness from AI Feedback — introduced
The model produced by finetuning a pretrained LM on constitutionally revised harmlessness responses plus helpfulness samples, serving as the initial policy for the RL stage.
SL-CAI is not really meant to be judged on its own; it exists mainly to hand the next training stage somewhere reasonable to start from. It comes from finetuning a plain language model on all those self-critiqued, self-revised responses, mixed with ordinary helpfulness examples so it doesn't go blunt along the way.
How this finetuned model gets produced, and its one downstream job, take up most of the page, which connects back to the critique-and-revision process supplying its data and forward to RL-CAI, the model it seeds.
The finetuned output of the supervised stage
SL-CAI is the model produced by finetuning a pretrained language model on the output of Critique and Revision: constitutionally revised harmlessness responses from all revision steps, mixed with separately collected helpfulness samples so the model does not lose helpfulness in the process (Critique and Revision — supplies the finetuning data for → SL-CAI (Supervised-Learning Constitutional AI model)). Training used 182,831 red-team prompts (four critique-revision pairs sampled per prompt from a Helpful RLHF Model) and 135,296 helpfulness prompts, for one epoch at a constant learning rate of 0.5 relative to the pretraining rate, batch size 1024 (constitutional-ai, §"3.2 Datasets and Training", p. 8). Variants are named by revision depth: SL-CAI-n is finetuned on data including up to the n-th revision, for n = 1, 2, 3, 4 (constitutional-ai, §"3.4 Scaling Trends", p. 9). Measured by Elo score, SL-CAI is more helpful and more harmless than a plain pretrained model, but less helpful than either RLHF baseline, and lands between Helpful RLHF Model and HH RLHF Model on harmlessness (constitutional-ai, §"3.3 Main Results", p. 8).
Its one job: seeding RL-CAI
SL-CAI's entire purpose within Constitutional AI (CAI) is to serve as the starting point for the second stage. It fills two roles at once for RL-CAI (RLAIF-trained Constitutional AI model): it is the initial policy snapshot the RL run begins from, and it is the model that generates the response pairs the Feedback Model labels (SL-CAI (Supervised-Learning Constitutional AI model) — seeds the policy and comparison data for → RL-CAI (RLAIF-trained Constitutional AI model)). The paper explains why both roles are given to the same model: "the distribution of responses generated by the policy are similar to the preference model training distribution, at least during early phases of RL" (constitutional-ai, §"4.1 Method", p. 11). Starting RL directly from a helpful-only RLHF model instead would leave the policy off-distribution relative to the AI-labeled comparisons it is being trained against — which is exactly the exploration cost the supervised stage was built to avoid. This doubled role is one instance of the The pipeline looks like a forward sequence; the causality doesn't run that way connective theme, and the handoff itself belongs to the Each pipeline stage's output is the next stage's engineered raw material connective theme alongside the analogous SFT-to-RM handoff in InstructGPT.