Critique and Revision
Constitutional AI: Harmlessness from AI Feedback — introduced
The supervised-stage pipeline in which a model critiques its own harmful response according to a sampled constitutional principle and then revises the response to remove the identified harm.
No human ever tells the model what is wrong with a harmful response. It is asked to critique its own output against one of the constitution's principles and then rewrite it in light of that critique, which is how Constitutional AI manufactures training data without a single human harm label.
The pipeline comes first, along with two checks on its design, before opening onto the few-shot prompts that steady it and the ablation testing whether the critique step actually earns its keep.
The supervised-stage pipeline
Critique and revision is the mechanism that produces harmlessness training data without any human harmfulness labels. Starting from a harmful response sampled from a Helpful RLHF Model, the model is asked to critique its own output against a randomly sampled principle from Constitution (set of principles), then to revise the response in light of that critique. The paper's own example: a request to help hack a neighbor's wifi, an initial response, a critique ("hacking into someone else's wifi is an invasion of their privacy and is possibly illegal"), and a revision that removes the harmful advice while explaining why (constitutional-ai, §"3.1 Method", p. 7). Because the revised prompt-response pair is formatted identically to the original, the pipeline can be applied repeatedly, producing a chain of progressively less harmful revisions — the paper reports the first revision "almost always removed most aspects of harmfulness," with later revisions improving results "less obvious[ly] by inspection" (constitutional-ai, §"3.1 Method", p. 8). Every revision from every step, not just the final one, becomes training data: the pipeline was run on 182,831 red-team prompts, four critique-revision pairs each, and the output finetunes a pretrained model into SL-CAI (Supervised-Learning Constitutional AI model) via the Critique and Revision — supplies the finetuning data for → SL-CAI (Supervised-Learning Constitutional AI model) edge.
Serving the whole method, and two design checks
As the supervised half of Constitutional AI (CAI), critique-and-revision exists to get the model "on-distribution" cheaply, reducing "the need for exploration and the total length of training during the second RL phase" (constitutional-ai, §"1.2 The Constitutional AI Approach", p. 5) — without it, RL-CAI (RLAIF-trained Constitutional AI model) would have to explore harmlessness behavior from a purely helpful starting point. Two children probe the design. Few-shot Prompts for SL-CAI (Appendix E.1) fixes a concrete bug: asking one model to play critic then reviser in the same context caused it to occasionally confuse the two roles, and hand-written worked examples resolve this. Direct Revision (critique-free ablation) asks whether the critique step earns its keep at all, finding a real but shrinking benefit as model scale grows, yet the paper keeps critiques anyway for the transparency a written rationale provides. The paper also reports, as an early qualitative signal that would later motivate RL-CAI (RLAIF-trained Constitutional AI model)'s redesign of the Principles for RL-CAI (Appendix C.2), that revised responses were "rarely evasive," in contrast to prior human-feedback-trained harmlessness models (constitutional-ai, §"3.1 Method", p. 8).