RL-CAI (RLAIF-trained Constitutional AI model)
Constitutional AI: Harmlessness from AI Feedback — introduced
The final policy produced by running RL against a preference model trained partly on AI-generated harmlessness labels, starting from the SL-CAI snapshot.
RL-CAI is Constitutional AI's answer to a question the rest of the paper spends its time setting up: what happens when reinforcement learning runs against a preference model partly taught by another AI rather than by people. It starts from the SL-CAI snapshot and pushes further.
The page presents it as the method's finished policy, then argues that it shifts a known tradeoff rather than merely sliding along it, connecting to evasiveness and the Pareto-improvement finding that follow from that claim.
The final policy of Constitutional AI
RL-CAI is the policy produced by running reinforcement learning against a preference model trained partly on AI-generated harmlessness labels, starting from the SL-CAI (Supervised-Learning Constitutional AI model) snapshot (SL-CAI (Supervised-Learning Constitutional AI model) — seeds the policy and comparison data for → RL-CAI (RLAIF-trained Constitutional AI model)). From this point the training pipeline is identical to ordinary RLHF, except the preference model is now the Hybrid Human/AI Preference Model that RL from AI Feedback (RLAIF) produces. The paper reports RL-CAI as "significantly more harmless than the RLHF and SL-CAI models" (constitutional-ai, §"4.3 Main Results", p. 12), with a chain-of-thought variant (RL-CAI w/ CoT) that trades a little helpfulness for a little more harmlessness. Left training too long, RL-CAI can be over-trained into Goodharting behavior — responding to harmful prompts with boilerplate reassurance ("you are valid, valued, and cared for") rather than substantive engagement (constitutional-ai, §"4.3 Main Results", p. 12), the paper's own explicit naming of the Goodhart's law pattern that the corpus otherwise only instantiates without labeling.
Breaking the tradeoff, not just moving along it
RL-CAI's headline result is read directly off where its Elo points fall relative to the line traced by Helpful RLHF Model and HH RLHF Model: ordinary RLHF only lets you trade helpfulness for harmlessness by moving along that line, while RL-CAI lands above it — at matched or better helpfulness, more harmless than either baseline (RL-CAI (RLAIF-trained Constitutional AI model) — produces → Pareto Improvement (helpfulness vs. harmlessness)). The mechanism is supply, not a cleverer RL algorithm: replacing scarce, expensive human harmfulness comparisons with abundant AI-generated ones lets far more harmlessness training happen without re-importing the evasiveness that saturated the earlier human-labeled data. Concretely, the paper reports RL-CAI is "virtually never evasive, and often gives nuanced and harmless responses to most red team prompts" (constitutional-ai, §"4.4 Harmlessness vs. Evasiveness", p. 13; RL-CAI (RLAIF-trained Constitutional AI model) — eliminates → Evasiveness), in direct contrast to HH RLHF's canned refusals. This is the empirical payoff the entire Constitutional AI's method: principles, critique, and revision pipeline was built to produce, and the One paper's central result, traced edge by edge connective theme traces exactly how redesigned principles and a redesigned Elo-scoring instruction, not a new RL algorithm, produced it.