← Explorean AI safety wiki themeHelp

Constitutional AI's method: principles, critique, and revision

Constitutional AI (2022) trains a harmless assistant from a short written list of principles instead of thousands of human harm judgments. This theme covers the mechanics of how that substitution actually works: a model critiques its own harmful response against a sampled principle, then revises the response to remove what it identified, and the result becomes the model that seeds everything downstream.

The critique-and-revision pipeline unfolds step by step, closing with the direct-revision ablation, which tests whether the critique step was ever necessary in the first place.

Constitutional AI trains a harmless assistant using only a small set of written principles as the source of human guidance, rather than human harmfulness labels, and this theme covers the mechanics of that method's supervised stage. The constitution is the set of natural-language principles steering the whole pipeline, split in practice into two specific lists: the 16 critique/revision instruction pairs used in the supervised stage, and a separate 16 multiple-choice comparison principles used later by the feedback model. Critique and revision is the pipeline itself, a model critiques its own harmful response according to a sampled principle, then revises the response to remove the identified harm, producing SL-CAI, the fine-tuned model that results and that seeds the later RL stage. Hand-written few-shot examples prevent the model from confusing the critique and revision roles it must switch between. Principle ensembling, sampling a different principle per comparison rather than reusing one throughout, and the direct-revision ablation, which tests whether the critique step is even necessary, both validate specific design choices within this pipeline rather than proposing new ones.

Members — a guided walk

  1. Constitutional AI (CAI)

    Constitutional AI opens this theme as the whole two-stage method: training a harmless assistant with only a small set of written principles as guidance, combining supervised self-critique and revision with reinforcement learning from AI-generated preference labels, in place of human harmfulness labels.

  2. Constitution (set of principles)

    The constitution is what makes constitutional-ai workable without human harm labels: rather than tens of thousands of preference labels, CAI substitutes only about ten simple natural-language principles, compressed enough to steer both the critique/revision stage and, later, the comparison stage.

  3. Principles for SL-CAI (Appendix C.1)

    Principles for SL-CAI is the constitution's first concrete expression: 16 critique/revision instruction pairs, phrased as imperatives addressed to a model acting on its own response, that steer the supervised stage this theme is about to walk through.

  4. Few-shot Prompts for SL-CAI (Appendix E.1)

    Few-shot prompts for SL-CAI fix a bug that critique-and-revision would otherwise hit: asking one model to play critic then reviser in sequence repeatedly confused its point of view, so hand-written worked examples of the pattern are prepended to keep the two roles straight before the real pipeline runs.

  5. Critique and Revision

    Critique and revision is the supervised pipeline itself, now equipped with sl-cai-principles to steer it and sl-cai-few-shot-examples to keep its roles from blurring: a model critiques its own harmful response according to a sampled principle, then revises the response to remove the identified harm.

  6. Direct Revision (critique-free ablation)

    Direct revision, the critique-free ablation, asks a pointed question about critique-and-revision's own design: is the critique step doing anything? At smaller scales critiqued revisions score more harmless, but the gap nearly closes at 52B, and the critiques were kept anyway for the human-readable record they provide.

  7. SL-CAI (Supervised-Learning Constitutional AI model)

    SL-CAI is not trained on any separately curated dataset; it is a pretrained model finetuned directly on critique-and-revision's output, run across 182,831 red-team prompts with four critique-revision pairs each, which is why SL-CAI variants are named by how many rounds of revision they were trained on.

  8. Principle Ensembling

    Principle ensembling, applied back at the revision step, samples a different one of sl-cai-principles for each revision instead of reusing one throughout; it barely moves harmlessness scores on its own, but the resulting diversity in revised responses is what pays off once those revisions seed the RL stage.

  9. Principles for RL-CAI (Appendix C.2)

    Principles for RL-CAI close this theme by mirroring sl-cai-principles in a different grammatical shape: the same 16-principle constitution re-expressed as compare-and-choose comparison questions rather than act-and-revise instructions, handing the constitution off to the feedback model's engineering that follows.

Superthemes

Edges explored within this theme

Super edges touching this theme

Sources

Every passage behind this theme’s member concepts.

13 source passages in 1 paper