Constitutional AI (CAI)
Constitutional AI: Harmlessness from AI Feedback — introduced
A two-stage method for training a harmless language model assistant using only a set of written principles rather than human harmfulness labels, combining supervised self-critique/revision with reinforcement learning from AI-generated preference labels.
Teaching a model to avoid harmful behavior normally means paying people to read and label thousands of harmful examples by hand. Constitutional AI replaces that labor with a short written list of principles, then has the model critique and revise its own output against them instead.
Two stages carry the method, laid out here along with where it sits in the wider corpus and what each stage contributes, opening onto the constitution, the critique-and-revision loop, and the two models that loop produces.
A two-stage alternative to human-labeled harmlessness
Constitutional AI (CAI) is a training method that produces a helpful and harmless assistant while replacing every human harmfulness label with a short written document. The paper states the design directly: "(Supervised Stage) Critique → Revision → Supervised Learning" followed by "(RL Stage) AI Comparison Evaluations → Preference Model → Reinforcement Learning" (constitutional-ai, §"1.2 The Constitutional AI Approach", p. 5). In the first stage, a Helpful RLHF Model generates a harmful response, critiques it against a sampled principle, and revises it; the Critique and Revision outputs finetune a pretrained model into SL-CAI (Supervised-Learning Constitutional AI model). In the second stage, RL from AI Feedback (RLAIF) uses that same set of principles to have a Feedback Model label which of two responses is more harmless, and RL against the resulting preference model produces RL-CAI (RLAIF-trained Constitutional AI model). Both stages draw their instructions from a single small Constitution (set of principles) rather than from a large private comparison dataset — "only of order ten simple principles, stated in natural language" replace "tens of thousands of human preference labels" (constitutional-ai, §"1.1 Motivations", p. 3; see Constitution (set of principles) — replaces human harm labels in → Constitutional AI (CAI)).
Where it sits in the corpus
CAI is the corpus's third and final move in the From an open problem to reward models to AI feedback: Concrete Problems names scalable oversight as an open problem in 2016, InstructGPT's Reward model (RM) operationalizes a scaled human-feedback solution via Reinforcement learning from human feedback (RLHF), and CAI pushes the same lineage one step further by having an AI, not a human, answer the harmlessness comparison question. It also answers a specific documented failure from Anthropic's own prior work: models trained with ordinary human-feedback RLHF for both helpfulness and harmlessness (HH RLHF Model) tend to become evasive, retreating into canned refusals. CAI is framed explicitly as an attempt to break that tension (the The helpfulness-harmlessness tradeoff and CAI's claim to beat it theme) without simply trading one property for the other, and its headline empirical claim, tracked through the One paper's central result, traced edge by edge, is that RL-CAI lands above the standard RLHF tradeoff curve rather than sliding along it. The method also instantiates a pattern that recurs at every level of the paper, captured in the Replacing the human in the loop at exactly the point it was the bottleneck connective theme: an AI process substitutes for a human one at exactly the point human throughput was scarce, while the surrounding machinery (preference-model training, PPO, the KL penalty) is left untouched.
The two stages and what each contributes
CAI's five direct children divide cleanly along its two stages. Constitution (set of principles) is the shared input, the natural-language principles that steer both stages. Critique and Revision and SL-CAI (Supervised-Learning Constitutional AI model) form the supervised stage: the pipeline itself, and the finetuned model it produces, whose job is to get the policy "on-distribution" before RL begins. RL from AI Feedback (RLAIF) and RL-CAI (RLAIF-trained Constitutional AI model) form the reinforcement-learning stage: the AI-feedback mechanism, and the final policy trained against a preference model that mixes AI-labeled harmlessness comparisons with human-labeled helpfulness comparisons. This grouping is itself part of the Constitutional AI's method: principles, critique, and revision theme. Every one of the eighteen concepts in this batch is a piece of this same machine — a principle list, a prompting fix, an ablation, or a labeling-format decision — which is why CAI, as the root node, is best understood not as one technique but as the sum of these parts operating in sequence.