Constitutional AI: Harmlessness from AI Feedback

arXiv:2212.08073 · PDF · 34 pp. · its place among the papers

One paper, several tellings. The big picture states the paper’s own thesis and contributions in its own terms; Inside the paper follows the paper’s own arc; Across the corpus traces how it connects to the other papers; The concepts lists everything the paper uses, in reading order. Pick a tab, then use the +/− toggles to open it level by level, or set a granularity to read the whole thing at that zoom.

Training a harmless assistant used to mean collecting tens of thousands of human labels on harmful conversations — and the assistants that resulted often learned to dodge uncomfortable questions rather than answer them well. Constitutional AI, from 2022, asks how far that human labeling can be replaced by something smaller and more legible: a written list of principles.

The chapters follow the paper's case: the two entangled problems of labeling cost and evasiveness, the supervised stage where a model critiques and revises its own responses, the reinforcement stage where an AI labeler replaces the human one, the evidence that the result improves on both axes at once, and the strains the method shows when pushed too hard.

Read at

Constitutional AI's argument: a page of principles can replace human harm labels (5 chapters)

The paper's thesis is that the human harmlessness labeler can be removed from RLHF and replaced by a short constitution of written principles — applied once to make a model critique and revise its own harmful outputs, and once to steer an AI feedback model that labels harmlessness preferences — and that the result is not a trade but a Pareto improvement: more harmless and less evasive at comparable helpfulness. The evidence is crowdworker Elo scores, absolute harm ratings, and ablations over the method's parts; the reported limits are Goodhart-style over-training and the human helpfulness labels the pipeline still consumes.

Chapter 1Two problems, one diagnosis (3 concepts)

The paper starts from a tension its own predecessors documented: pushing helpfulness raises harm, and pushing harmlessness breeds evasion — canned refusals where engagement was wanted. Beneath the tension sits a scaling problem: harmlessness training ran on large volumes of human labels, which are expensive, slow, and opaque as a specification of values. The paper's diagnosis is that both problems share a fix — move the harm judgment into the model, under explicit written control.

The documented tradeoff concept (2 connections)

Helpfulness and harmlessness pull against each other in standard RLHF training — the empirical regularity the paper sets out to break rather than ride.

connections

Helpfulness-Harmlessness Tension — collapses into → Evasiveness
Refusal (as a steerable behavior) — quantifies → Helpfulness-Harmlessness Tension hindsight

The failure it will not accept concept (7 connections)

'I can't answer that' as a learned reflex: apparent safety bought with transparency and helpfulness, which the paper treats as a failure mode, not a success.

connections

Elo Score (helpfulness/harmlessness) — was tuned to penalize → Evasiveness
Excessive hedging on simple questions — is the harmlessness axis mirror of → Evasiveness hindsight
Goodhart's law — is instantiated through mislabeled proxy in → Evasiveness hindsight
Helpfulness-Harmlessness Tension — collapses into → Evasiveness
Per-token KL penalty from the SFT model — was not the fix applied to → Evasiveness hindsight
Refusal (as a steerable behavior) — generalizes → Evasiveness hindsight
RL-CAI (RLAIF-trained Constitutional AI model) — eliminates → Evasiveness

Oversight, taken to an extreme concept (4 connections)

The paper presents itself as an extreme instance of scalable oversight: AI systems helping humans supervise other AIs, with human guidance compressed into the principles themselves.

connections

Scalable oversight — is addressed by adapting → Distant supervision
Scalable oversight — is addressed by → Hierarchical reinforcement learning (HRL)
Scalable oversight — is retroactively operationalized by → Reward model (RM) hindsight
Scalable oversight — is operationalized as → Semi-supervised reinforcement learning

Chapter 2Stage one: the model revises itself (4 concepts)

The supervised stage turns a helpful-only model against its own outputs. Prompted with red-team questions, it produces harmful responses; then, steered by principles sampled from the constitution, it critiques each response and rewrites it to remove the identified harm. Fine-tuning on the revisions yields a model already substantially less harmful — before any reinforcement learning — with every step of the correction legible on paper.

The method, whole concept (1 connection)

Two stages — supervised self-revision, then RL from AI feedback — with a set of written principles as the only source of harmlessness guidance in either.

connections

Constitution (set of principles) — replaces human harm labels in → Constitutional AI (CAI)

The constitution concept (1 connection)

A small set of natural-language principles — the entire human specification of harmlessness, short enough to read and to argue with.

connections

Constitution (set of principles) — replaces human harm labels in → Constitutional AI (CAI)

Critique, then revise concept (3 connections)

The model names what is wrong with its response under a sampled principle, then rewrites it — self-correction as a data-generation pipeline.

connections

Direct Revision (critique-free ablation) — tests the necessity of critique in → Critique and Revision
Few-shot Prompts for SL-CAI (Appendix E.1) — prevents role confusion in → Critique and Revision
Critique and Revision — supplies the finetuning data for → SL-CAI (Supervised-Learning Constitutional AI model)

The supervised result concept (2 connections)

Fine-tuning on revised responses (plus helpfulness samples) gives SL-CAI — the starting policy the RL stage then sharpens.

connections

Critique and Revision — supplies the finetuning data for → SL-CAI (Supervised-Learning Constitutional AI model)
SL-CAI (Supervised-Learning Constitutional AI model) — seeds the policy and comparison data for → RL-CAI (RLAIF-trained Constitutional AI model)

Chapter 3Stage two: AI feedback replaces the labeler (4 concepts)

The reinforcement stage keeps the RLHF pipeline intact and swaps its data source. A feedback model — shown a harmful prompt, two candidate responses, and a constitutional principle in multiple-choice format — produces the harmlessness preference labels, made more accurate and more transparent by chain-of-thought reasoning. A hybrid preference model trains on human helpfulness labels and AI harmlessness labels together, and RL against it produces the paper's final model.

RLHF minus the H concept (4 connections)

RL from AI feedback: the established pipeline with the human harmlessness labeler replaced by a model — the paper's structural contribution, and its name for what changed.

connections

RL from AI Feedback (RLAIF) — delegates harmlessness comparisons to → Feedback Model
RL from AI Feedback (RLAIF) — makes newly automatable → Iterated (Online) Training
Open-ended generation evaluation with GPT-4 rating — shares the ai judge move with → RL from AI Feedback (RLAIF) hindsight
Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)

The new labeler concept (4 connections)

An independent language model whose multiple-choice probabilities become preference labels — supervision generated at model speed and model cost.

connections

Feedback Model — supplies the harmlessness half of → Hybrid Human/AI Preference Model
Multiple-Choice Comparison Format — turns into a preference labeler → Feedback Model
Principle Ensembling — makes more robust → Feedback Model
RL from AI Feedback (RLAIF) — delegates harmlessness comparisons to → Feedback Model

Judgments that show their work concept (1 connection)

Asking the feedback model to reason step by step before choosing makes its harm judgments both more accurate and more inspectable.

connections

Chain-of-Thought (CoT) Prompting — creates the overconfidence that necessitates → Preference Label Clamping (Soft vs. Hard vs. Clamped)

The final model concept (3 connections)

RL-CAI: the policy trained against the hybrid preference model, starting from SL-CAI — the model all the paper's headline comparisons are about.

connections

RL-CAI (RLAIF-trained Constitutional AI model) — eliminates → Evasiveness
RL-CAI (RLAIF-trained Constitutional AI model) — produces → Pareto Improvement (helpfulness vs. harmlessness)
SL-CAI (Supervised-Learning Constitutional AI model) — seeds the policy and comparison data for → RL-CAI (RLAIF-trained Constitutional AI model)

Chapter 4The evidence: better on both axes (4 concepts)

The evaluation asks crowdworkers to compare models in conversation and converts their preferences into Elo scores for helpfulness and harmlessness separately. Against the human-feedback baseline, RL-CAI rates as more harmless at comparable helpfulness — the frontier moves outward rather than sliding along the old tradeoff, which is the paper's headline finding. Absolute harm ratings corroborate the pairwise picture, and ablations test the parts, including whether the critique step of the supervised stage is necessary at all.

The measuring stick concept (4 connections)

Crowdworker pairwise preferences, converted to Elo ratings per axis, track helpfulness and harmlessness across model snapshots throughout training.

connections

Absolute Harmfulness Score — cross checks → Elo Score (helpfulness/harmlessness)
Elo Score (helpfulness/harmlessness) — supplies the explanatory analogy for → Bradley-Terry Model
Elo Score (helpfulness/harmlessness) — was tuned to penalize → Evasiveness
Elo Score (helpfulness/harmlessness) — generalizes → Win rate (preference against baseline) hindsight

The headline finding concept (1 connection)

Constitutional RL shifts the helpfulness-harmlessness frontier outward — better on one axis without paying on the other — rather than trading along it.

connections

RL-CAI (RLAIF-trained Constitutional AI model) — produces → Pareto Improvement (helpfulness vs. harmlessness)

The second instrument concept (1 connection)

A 0-4 harmfulness score on single conversations, fit to crowdworker ratings — an absolute check on what the relative Elo comparisons say.

connections

Absolute Harmfulness Score — cross checks → Elo Score (helpfulness/harmlessness)

Is the critique needed? concept (1 connection)

An ablation that skips the critique and revises directly — the paper's way of testing which stage-one step carries the harm reduction.

connections

Direct Revision (critique-free ablation) — tests the necessity of critique in → Critique and Revision

Chapter 5Where it strains (3 concepts)

The paper reports the method's edges. Trained too long against its preference model, RL-CAI turns harsh and accusatory toward harmless-seeming prompts — Goodhart's law surfacing inside the very pipeline built to encode judgment, managed with countermeasures like label clamping rather than eliminated. And the replacement of humans is partial by design: helpfulness labels are still collected from people, and the whole scheme leans on the feedback model's probabilities being trustworthy.

Goodhart, inside the fix concept (4 connections)

Over-optimizing against the preference model degrades true response quality while the proxy reward climbs — the corpus's oldest failure mode, observed in its newest method.

connections

Goodhart's law — is instantiated through mislabeled proxy in → Evasiveness hindsight
Goodhart's law — operates at the subfeature level in → Excessive hedging on simple questions hindsight
Goodhart's law — is empirically instantiated by → Reward model over-optimization hindsight
Sycophancy — instantiates → Goodhart's law hindsight

The human data still inside concept (1 connection)

The pipeline starts from a helpful RLHF model and keeps training on human helpfulness comparisons — the constitution replaces harm labels, not human feedback wholesale.

connections

Helpful RLHF Model — traces the tradeoff curve with → HH RLHF Model

A load-bearing assumption concept (2 connections)

Treating the feedback model's output probabilities as soft preference labels works only because those probabilities are reasonably calibrated — a property borrowed from prior work and relied on throughout.

connections

Model Calibration (well-calibrated multiple-choice probabilities) — is empirically verified on → HHH Evaluation Set (binary comparisons)
Model Calibration (well-calibrated multiple-choice probabilities) — is the precondition for → Multiple-Choice Comparison Format

By 2022 the recipe for training a helpful assistant was settled: collect human preferences, fit a preference model, let reinforcement learning optimize against it. But the harmlessness half of those preferences was producing assistants that dodged hard questions instead of answering them, and every fix seemed to require more human labels. This paper asks whether the harmlessness labels can be replaced by something smaller and more legible: a short list of written principles.

The chapters follow the paper's own order: the tension that motivates it, the supervised stage where the model critiques and rewrites its own worst answers, the RL stage where a model supplies the preference labels, and the measurements behind the claim that the tradeoff between helping and refusing actually moved.

Read at

Inside Constitutional AI: harmlessness from a list of principles (4 chapters)

Constitutional AI trains a harmless assistant with no human harmlessness labels at all: a constitution of written principles steers the model to critique and revise its own harmful outputs in the supervised stage, then steers a feedback model to label harmlessness preferences in the RL stage — and the resulting RL-CAI model is rated both more harmless and less evasive than models trained on human harm labels.

Chapter 1The problem: harmlessness that learns to hide (5 sections)

The paper opens on a failure of its authors' own prior work: reinforcement learning from human feedback can make an assistant harmless, but the cheapest route to harmlessness is refusal, and crowdworkers rewarding harmlessness in isolation had produced models that met sensitive questions with boilerplate (constitutional-ai, §"1.1 Motivations", p. 2). The motivations section frames the fix as a supervision-scaling problem: as capabilities grow, human oversight becomes the bottleneck, so let AI systems carry part of it (constitutional-ai, §"Scaling Supervision", p. 2). Before staking anything on that idea, the paper checks its feasibility — language models can already tell helpful, honest, and harmless responses apart, and get better at it with scale. The one input no principle can generate, harmful prompts, comes from human red teaming, the same crowdworker machinery the rest of the paper works to move beyond.

The tension the paper starts from concept (2 connections)

Helpfulness training and harmlessness training pull against each other: assistants trained to help more will help with harm, and assistants trained against harm learn to disengage. The whole paper is an attempt to show this tradeoff is not fixed (constitutional-ai, §"1.1 Motivations", p. 2).

connections

Helpfulness-Harmlessness Tension — collapses into → Evasiveness
Refusal (as a steerable behavior) — quantifies → Helpfulness-Harmlessness Tension hindsight

Evasiveness, the named failure concept (7 connections)

The paper gives the failure a name: canned refusals that score as harmless while giving the user nothing. Naming it lets the paper measure it, and later tune its Elo evaluations to penalize it (constitutional-ai, §"A Harmless but Non-Evasive (Still Helpful) Assistant", p. 4).

connections

Elo Score (helpfulness/harmlessness) — was tuned to penalize → Evasiveness
Excessive hedging on simple questions — is the harmlessness axis mirror of → Evasiveness hindsight
Goodhart's law — is instantiated through mislabeled proxy in → Evasiveness hindsight
Helpfulness-Harmlessness Tension — collapses into → Evasiveness
Per-token KL penalty from the SFT model — was not the fix applied to → Evasiveness hindsight
Refusal (as a steerable behavior) — generalizes → Evasiveness hindsight
RL-CAI (RLAIF-trained Constitutional AI model) — eliminates → Evasiveness

Scaling supervision concept (4 connections)

Inherited from the 2016 agenda as an open problem, scalable oversight acquires an operational reading here: use AI to supervise AI at exactly the points where human labeling cannot keep up (constitutional-ai, §"Scaling Supervision", p. 2).

connections

Scalable oversight — is addressed by adapting → Distant supervision
Scalable oversight — is addressed by → Hierarchical reinforcement learning (HRL)
Scalable oversight — is retroactively operationalized by → Reward model (RM) hindsight
Scalable oversight — is operationalized as → Semi-supervised reinforcement learning

First, check that the judge can judge concept (3 connections)

Before delegating harmlessness judgments to a model, the paper verifies that models can make them: on binary helpful-honest-harmless comparisons, identification accuracy climbs steadily with model size (constitutional-ai, §"2 Evaluating the Potential for AI Supervision of HHH", p. 6).

connections

HHH Evaluation Set (binary comparisons) — is hosted as a subset of → BIG-Bench
Helpful, honest, harmless (HHH) alignment framework — is operationalized as → HHH Evaluation Set (binary comparisons)
Model Calibration (well-calibrated multiple-choice probabilities) — is empirically verified on → HHH Evaluation Set (binary comparisons)

Where the harmful prompts come from theme (5 concepts)

Every stage downstream trains or evaluates on adversarial prompts gathered by human red teaming; this theme covers that sourcing, the appendix evaluations built from the same transcripts, and the automated version the paper points toward.

Red Teaming (5 connections)
- Red Teaming — is the manual process that scales into → Automated Red Teaming
- CAA as an adversarial red-teaming tool (proposed application) — relocates the attack surface of → Red Teaming hindsight
- Red Teaming — supplies the transcripts for → Harmful Behavior Identification/Classification Evaluations (Appendix B)
- Jailbreaks — is the target phenomenon of → Red Teaming
- PALMS (Process for Adapting Language Models to Society) — provides an independent check outside → Red Teaming
Red teaming opens this theme as the human-driven source of every harmful prompt in the pipeline: crowdworkers hold adversarial conversations designed to bait the assistant into harmful outputs, and this transcript supply feeds both SL-CAI and RL-CAI training directly.
Jailbreaks (2 connections)
- Jailbreaks — motivates → CAA as an adversarial red-teaming tool (proposed application)
- Jailbreaks — is the target phenomenon of → Red Teaming
Jailbreaks name the problem red teaming exists to get ahead of: adversarial or unusual inputs ordinary users find on their own, without any crowdworker process, that make a model output content its safety training was meant to prevent. Steering Llama 2 via Contrastive Activation Addition (2023) cites them as the reason ad hoc, deployment-time discovery of unwanted behavior is unreliable, proposing its own steering vectors as a systematic alternative for eliciting the same failure under controlled conditions before deployment.
Harmful Behavior Identification/Classification Evaluations (Appendix B) (1 connection)
- Red Teaming — supplies the transcripts for → Harmful Behavior Identification/Classification Evaluations (Appendix B)
Harmful behavior identification and classification evaluations are not independently written; both of their multiple-choice tasks are built directly out of red-teaming transcripts and the crowdworker harm ratings collected alongside them, testing whether a model can replicate those same judgments.
Automated Red Teaming (2 connections)
- CAA as an adversarial red-teaming tool (proposed application) — skips the prompt search of → Automated Red Teaming hindsight
- Red Teaming — is the manual process that scales into → Automated Red Teaming
Automated red teaming looks past the crowdworker labor ceiling that red teaming runs into, proposing a language model generate the adversarial prompts itself; the paper notes this scaling move depends on non-evasiveness, since a model trained to simply refuse would teach an automated red-teamer nothing.
PALMS (Process for Adapting Language Models to Society) (1 connection)
- PALMS (Process for Adapting Language Models to Society) — provides an independent check outside → Red Teaming
PALMS closes the theme by stepping outside the loop entirely: every harmful prompt discussed so far, real or automated, traces back to red-teaming, but PALMS is an externally authored, fixed list of sensitive questions never part of that data, serving as an out-of-distribution spot check on generalization.

Chapter 2Act one: the model rewrites its own answers (6 sections)

The supervised stage deliberately starts from the most dangerous model available: a helpful-only RLHF assistant that responds to red-team prompts with genuinely harmful content. Each such response is handed back to the model along with a principle sampled from the constitution — first critique your answer against the principle, then revise it to remove the harm, possibly several times over (constitutional-ai, §"3 Constitutional AI: Critiques, Revisions, and Supervised Learning", p. 7). A pretrained model finetuned on the revisions, with helpfulness samples mixed in so it does not forget how to answer, becomes SL-CAI: already less harmful, and close enough to the target behavior that the RL stage will not have to travel far. An ablation asks whether the critique step earns its keep, and the answer turns out to depend on scale.

The method, gathered in one place theme (9 concepts)

The theme walks the whole supervised machinery: the constitution's two principle sets, the few-shot examples that keep the critique and revision roles from blurring, and the ablations behind each design choice.

Constitutional AI (CAI) (1 connection)
- Constitution (set of principles) — replaces human harm labels in → Constitutional AI (CAI)
Constitutional AI opens this theme as the whole two-stage method: training a harmless assistant with only a small set of written principles as guidance, combining supervised self-critique and revision with reinforcement learning from AI-generated preference labels, in place of human harmfulness labels.
Constitution (set of principles) (1 connection)
- Constitution (set of principles) — replaces human harm labels in → Constitutional AI (CAI)
The constitution is what makes constitutional-ai workable without human harm labels: rather than tens of thousands of preference labels, CAI substitutes only about ten simple natural-language principles, compressed enough to steer both the critique/revision stage and, later, the comparison stage.
Principles for SL-CAI (Appendix C.1) (2 connections)
- Principle Ensembling — diversifies use of → Principles for SL-CAI (Appendix C.1)
- Principles for SL-CAI (Appendix C.1) — is mirrored in a different shape by → Principles for RL-CAI (Appendix C.2)
Principles for SL-CAI is the constitution's first concrete expression: 16 critique/revision instruction pairs, phrased as imperatives addressed to a model acting on its own response, that steer the supervised stage this theme is about to walk through.
Few-shot Prompts for SL-CAI (Appendix E.1) (1 connection)
- Few-shot Prompts for SL-CAI (Appendix E.1) — prevents role confusion in → Critique and Revision
Few-shot prompts for SL-CAI fix a bug that critique-and-revision would otherwise hit: asking one model to play critic then reviser in sequence repeatedly confused its point of view, so hand-written worked examples of the pattern are prepended to keep the two roles straight before the real pipeline runs.
Critique and Revision (3 connections)
- Direct Revision (critique-free ablation) — tests the necessity of critique in → Critique and Revision
- Few-shot Prompts for SL-CAI (Appendix E.1) — prevents role confusion in → Critique and Revision
- Critique and Revision — supplies the finetuning data for → SL-CAI (Supervised-Learning Constitutional AI model)
Critique and revision is the supervised pipeline itself, now equipped with sl-cai-principles to steer it and sl-cai-few-shot-examples to keep its roles from blurring: a model critiques its own harmful response according to a sampled principle, then revises the response to remove the identified harm.
Direct Revision (critique-free ablation) (1 connection)
- Direct Revision (critique-free ablation) — tests the necessity of critique in → Critique and Revision
Direct revision, the critique-free ablation, asks a pointed question about critique-and-revision's own design: is the critique step doing anything? At smaller scales critiqued revisions score more harmless, but the gap nearly closes at 52B, and the critiques were kept anyway for the human-readable record they provide.
SL-CAI (Supervised-Learning Constitutional AI model) (2 connections)
- Critique and Revision — supplies the finetuning data for → SL-CAI (Supervised-Learning Constitutional AI model)
- SL-CAI (Supervised-Learning Constitutional AI model) — seeds the policy and comparison data for → RL-CAI (RLAIF-trained Constitutional AI model)
SL-CAI is not trained on any separately curated dataset; it is a pretrained model finetuned directly on critique-and-revision's output, run across 182,831 red-team prompts with four critique-revision pairs each, which is why SL-CAI variants are named by how many rounds of revision they were trained on.
Principle Ensembling (2 connections)
- Principle Ensembling — makes more robust → Feedback Model
- Principle Ensembling — diversifies use of → Principles for SL-CAI (Appendix C.1)
Principle ensembling, applied back at the revision step, samples a different one of sl-cai-principles for each revision instead of reusing one throughout; it barely moves harmlessness scores on its own, but the resulting diversity in revised responses is what pays off once those revisions seed the RL stage.
Principles for RL-CAI (Appendix C.2) (1 connection)
- Principles for SL-CAI (Appendix C.1) — is mirrored in a different shape by → Principles for RL-CAI (Appendix C.2)
Principles for RL-CAI close this theme by mirroring sl-cai-principles in a different grammatical shape: the same 16-principle constitution re-expressed as compare-and-choose comparison questions rather than act-and-revise instructions, handing the constitution off to the feedback model's engineering that follows.

Start from the helpful, harmful model concept (1 connection)

The pipeline begins from a helpful-only RLHF model precisely because it answers red-team prompts with usable harm — you cannot revise a refusal (constitutional-ai, §"1.4 Models and Data", p. 6).

connections

Helpful RLHF Model — traces the tradeoff curve with → HH RLHF Model

The constitution concept (1 connection)

Sixteen critique-revision instruction pairs for the supervised stage (Principles for SL-CAI (Appendix C.1)) and sixteen comparison principles for RL (Principles for RL-CAI (Appendix C.2)): the entire human contribution to harmlessness, small enough to read in minutes (constitutional-ai, §"1.2 The Constitutional AI Approach", p. 5).

connections

Constitution (set of principles) — replaces human harm labels in → Constitutional AI (CAI)

Critique, then revise concept (3 connections)

The loop at the heart of the supervised stage: the model states how its response violates a sampled principle, then rewrites the response to remove the harm; revisions can be chained, though most of the benefit arrives with the first.

connections

Direct Revision (critique-free ablation) — tests the necessity of critique in → Critique and Revision
Few-shot Prompts for SL-CAI (Appendix E.1) — prevents role confusion in → Critique and Revision
Critique and Revision — supplies the finetuning data for → SL-CAI (Supervised-Learning Constitutional AI model)

SL-CAI, the halfway model concept (2 connections)

Finetuning on revised responses plus helpfulness samples produces SL-CAI — not the finished model but the launchpad, built so that RL starts near the target distribution (constitutional-ai, §"3.2 Datasets and Training", p. 8).

connections

Critique and Revision — supplies the finetuning data for → SL-CAI (Supervised-Learning Constitutional AI model)
SL-CAI (Supervised-Learning Constitutional AI model) — seeds the policy and comparison data for → RL-CAI (RLAIF-trained Constitutional AI model)

Are critiques necessary? concept (1 connection)

Skipping the critique and revising directly works nearly as well at 52B parameters but measurably worse at smaller scales; the paper keeps the critique for transparency as much as for performance (constitutional-ai, §"3.5 Are Critiques Necessary?", p. 10).

connections

Direct Revision (critique-free ablation) — tests the necessity of critique in → Critique and Revision

Chapter 3Act two: preference labels without a human labeler (6 sections)

The RL stage is deliberately unoriginal: the same preference-model-plus-RL recipe as standard RLHF, with one substitution — the harmlessness comparisons in the preference-model training set are labeled by a model instead of by crowdworkers (constitutional-ai, §"4 Constitutional AI: Reinforcement Learning from AI Feedback", p. 10). Making a plain pretrained language model into a trustworthy labeler is the chapter's real work: a multiple-choice format whose answer probabilities become soft labels, a freshly sampled principle for each comparison (principle ensembling), chain-of-thought reasoning that improves the judgments but overshoots into overconfidence, and clamped labels to pull it back. Helpfulness labels stay human; the resulting hybrid preference model rewards a policy that begins as SL-CAI and ends as RL-CAI.

Engineering the labeler theme (8 concepts)

The full chain of prompting and calibration choices — the comparison format, the step-by-step trigger, hand-written chain-of-thought prompts, and the calibration the whole scheme leans on — that turns log-probabilities into preference labels.

Model Calibration (well-calibrated multiple-choice probabilities) (2 connections)
- Model Calibration (well-calibrated multiple-choice probabilities) — is empirically verified on → HHH Evaluation Set (binary comparisons)
- Model Calibration (well-calibrated multiple-choice probabilities) — is the precondition for → Multiple-Choice Comparison Format
Model calibration opens this theme as the empirical precondition everything else depends on: prior work established that language models' probabilities on multiple-choice questions are reasonably trustworthy, which is what licenses treating a plain LM's raw output as usable preference signal at all.
Multiple-Choice Comparison Format (3 connections)
- Multiple-Choice Comparison Format — turns into a preference labeler → Feedback Model
- Model Calibration (well-calibrated multiple-choice probabilities) — is the precondition for → Multiple-Choice Comparison Format
- Multiple-choice contrast-pair format — repurposes the ab mechanic of → Multiple-Choice Comparison Format hindsight
The multiple-choice comparison format is where that calibration gets put to work: it packages a conversation, a sampled constitutional principle, and two labeled candidate responses into an (A)/(B) question, so a model's ordinary next-token log-probabilities become a scalar preference label instead of noise.
Feedback Model (4 connections)
- Feedback Model — supplies the harmlessness half of → Hybrid Human/AI Preference Model
- Multiple-Choice Comparison Format — turns into a preference labeler → Feedback Model
- Principle Ensembling — makes more robust → Feedback Model
- RL from AI Feedback (RLAIF) — delegates harmlessness comparisons to → Feedback Model
Feedback model is the entity this format actually manufactures: on its own an independent pretrained LM has no built-in notion of preference, but the multiple-choice template is the specific engineering move that turns it into a preference labeler standing in for a crowdworker.
Chain-of-Thought (CoT) Prompting (1 connection)
- Chain-of-Thought (CoT) Prompting — creates the overconfidence that necessitates → Preference Label Clamping (Soft vs. Hard vs. Clamped)
Chain-of-thought prompting is layered onto the feedback model's judgment to make it more accurate and transparent, asking it to reason step by step before committing to the (A)/(B) answer that multiple-choice-evaluation-format extracts from it.
"Let's think step-by-step" zero-shot CoT trigger (1 connection)
- "Let's think step-by-step" zero-shot CoT trigger — is paired with → Chain-of-Thought Prompts for RL-CAI (Appendix E.2)
The 'let's think step-by-step' trigger is the zero-shot instrument that starts this reasoning: a single phrase borrowed from Kojima et al. that signals reasoning should happen before an answer, though it says nothing yet about what that reasoning should contain.
Chain-of-Thought Prompts for RL-CAI (Appendix E.2) (1 connection)
- "Let's think step-by-step" zero-shot CoT trigger — is paired with → Chain-of-Thought Prompts for RL-CAI (Appendix E.2)
Chain-of-thought prompts for RL-CAI supply what the trigger phrase alone cannot: hand-written few-shot examples with a substitutable principle placeholder and a worked reasoning chain, giving the feedback model a concrete template for what a harmlessness judgment's reasoning should look like.
Preference Label Clamping (Soft vs. Hard vs. Clamped) (1 connection)
- Chain-of-Thought (CoT) Prompting — creates the overconfidence that necessitates → Preference Label Clamping (Soft vs. Hard vs. Clamped)
Label clamping exists because chain-of-thought reasoning creates a side effect it must correct: a CoT explanation almost always states its preferred answer before the model outputs a probability, pushing that probability toward 0 or 1, so clamping forces it back into a fixed band such as 40-60%.
Principle Ensembling (2 connections)
- Principle Ensembling — makes more robust → Feedback Model
- Principle Ensembling — diversifies use of → Principles for SL-CAI (Appendix C.1)
Principle ensembling closes the theme by hardening the feedback model from a different angle: sampling a different RL-CAI principle for every comparison label rather than reusing one, which the paper reports led to notably more robust preference-model behavior than a fixed principle would.

RLAIF, the substitution named concept (4 connections)

The paper's most exportable idea, stated in its first paragraph: RLHF with the human preference labeler replaced by an AI one, for the harmlessness half of the data (constitutional-ai, §"Abstract", p. 1).

connections

RL from AI Feedback (RLAIF) — delegates harmlessness comparisons to → Feedback Model
RL from AI Feedback (RLAIF) — makes newly automatable → Iterated (Online) Training
Open-ended generation evaluation with GPT-4 rating — shares the ai judge move with → RL from AI Feedback (RLAIF) hindsight
Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)

The feedback model concept (4 connections)

A pretrained model, not a finetuned specialist: shown the conversation, a principle, and two candidate responses, its normalized answer probabilities become the label (constitutional-ai, §"4.1 Method", p. 10).

connections

Feedback Model — supplies the harmlessness half of → Hybrid Human/AI Preference Model
Multiple-Choice Comparison Format — turns into a preference labeler → Feedback Model
Principle Ensembling — makes more robust → Feedback Model
RL from AI Feedback (RLAIF) — delegates harmlessness comparisons to → Feedback Model

The A-or-B question concept (3 connections)

The load-bearing interface: framing the harmlessness judgment as a multiple-choice question is what lets calibrated log-probabilities stand in for a crowdworker's click (constitutional-ai, §"4.1 Method", p. 10).

connections

Multiple-Choice Comparison Format — turns into a preference labeler → Feedback Model
Model Calibration (well-calibrated multiple-choice probabilities) — is the precondition for → Multiple-Choice Comparison Format
Multiple-choice contrast-pair format — repurposes the ab mechanic of → Multiple-Choice Comparison Format hindsight

A hybrid preference model concept (1 connection)

Human helpfulness comparisons and AI harmlessness comparisons train a single preference model — the pipeline's quiet acknowledgment that only one of its two label streams has been automated (constitutional-ai, §"1.2 The Constitutional AI Approach", p. 5).

connections

Feedback Model — supplies the harmlessness half of → Hybrid Human/AI Preference Model

RL-CAI, the finished model concept (3 connections)

The final policy: RL against the hybrid preference model, starting from the SL-CAI snapshot. Everything the paper claims is measured on this model and its chain-of-thought variant (constitutional-ai, §"4 Constitutional AI: Reinforcement Learning from AI Feedback", p. 10).

connections

RL-CAI (RLAIF-trained Constitutional AI model) — eliminates → Evasiveness
RL-CAI (RLAIF-trained Constitutional AI model) — produces → Pareto Improvement (helpfulness vs. harmlessness)
SL-CAI (Supervised-Learning Constitutional AI model) — seeds the policy and comparison data for → RL-CAI (RLAIF-trained Constitutional AI model)

Chapter 4The verdict: a moved frontier, and a familiar failure (6 sections)

The headline result is plotted as crowdworker Elo scores: RL-CAI is rated as harmless as the human-feedback HH model while staying markedly more helpful, which the paper reads as a Pareto improvement — the frontier moved outward, not a new point on the old curve (constitutional-ai, §"1.1 Motivations", p. 3). The result is built as much from redesigning what gets rewarded and scored as from any change to the algorithm, an arc the wiki traces edge by edge in one connective thread. A second instrument, the absolute harmfulness score, cross-checks the comparisons on an independent scale. The paper is also candid about what over-optimizing its own reward does — trained too long, RL-CAI turns boilerplate and accusatory, Goodhart's law surfacing inside the very pipeline built to escape a different Goodhart failure (constitutional-ai, §"4.3 Main Results", p. 12). The future-work list closes the loop: with labels this cheap, keeping the preference model fresh stops being bottlenecked on people.

The tradeoff, measured theme (8 concepts)

The theme assembles the full empirical apparatus behind the paper's central claim: the two RLHF baselines, the Elo curves, and where RL-CAI lands relative to both.

Helpfulness-Harmlessness Tension (2 connections)
- Helpfulness-Harmlessness Tension — collapses into → Evasiveness
- Refusal (as a steerable behavior) — quantifies → Helpfulness-Harmlessness Tension hindsight
Constitutional AI's central empirical argument concerns a documented tradeoff: increasing a model's willingness to help tends to increase its potential for harm, while increasing harmlessness training tends to make a model more evasive rather than genuinely safer.
Evasiveness (7 connections)
- Elo Score (helpfulness/harmlessness) — was tuned to penalize → Evasiveness
- Excessive hedging on simple questions — is the harmlessness axis mirror of → Evasiveness hindsight
- Goodhart's law — is instantiated through mislabeled proxy in → Evasiveness hindsight
- Helpfulness-Harmlessness Tension — collapses into → Evasiveness
- Per-token KL penalty from the SFT model — was not the fix applied to → Evasiveness hindsight
- Refusal (as a steerable behavior) — generalizes → Evasiveness hindsight
- RL-CAI (RLAIF-trained Constitutional AI model) — eliminates → Evasiveness
That tension collapses concretely into evasiveness: when feedback rewards harmlessness without also penalizing refusal, the cheapest way for a model to score well on harmful prompts is to disengage with a canned "I can't answer that" rather than think the request through, exactly what Bai et al.'s prior HH RLHF training exhibited.
Refusal (as a steerable behavior) (3 connections)
- Custom refusal contrastive dataset — supplies contrast pairs for → Refusal (as a steerable behavior)
- Refusal (as a steerable behavior) — generalizes → Evasiveness hindsight
- Refusal (as a steerable behavior) — quantifies → Helpfulness-Harmlessness Tension hindsight
Refusal gives the same tension a direct dial, one Steering Llama 2 via Contrastive Activation Addition (2023) turns rather than just observes: a custom contrastive dataset spans both sensitive and entirely benign questions, broader than evasiveness's narrower focus on canned refusals to sensitive queries specifically. Moved up or down with a signed multiplier, refusal turns the helpful-versus-harmless tension from a property observed across differently trained models, like the two RLHF baselines below, into a continuous quantity measurable within a single model.
Helpful RLHF Model (1 connection)
- Helpful RLHF Model — traces the tradeoff curve with → HH RLHF Model
The helpful RLHF model is one of two baselines the paper uses to trace that tradeoff empirically: a policy trained on helpfulness comparisons only, serving as the harmful-response-generating starting point for the SL-CAI critique-and-revision pipeline.
HH RLHF Model (1 connection)
- Helpful RLHF Model — traces the tradeoff curve with → HH RLHF Model
The HH RLHF model differs from it in exactly one respect, harmlessness comparisons added to the same pipeline, and that single difference is what traces the tradeoff curve empirically: the helpful model is more helpful but more harmful, the HH model less harmful but more evasive, plotting two ends of the same line.
Elo Score (helpfulness/harmlessness) (4 connections)
- Absolute Harmfulness Score — cross checks → Elo Score (helpfulness/harmlessness)
- Elo Score (helpfulness/harmlessness) — supplies the explanatory analogy for → Bradley-Terry Model
- Elo Score (helpfulness/harmlessness) — was tuned to penalize → Evasiveness
- Elo Score (helpfulness/harmlessness) — generalizes → Win rate (preference against baseline) hindsight
The Elo score is the instrument used to plot that line, a rating computed from crowdworker pairwise comparisons tracking relative helpfulness and harmlessness across training snapshots. It is not a neutral instrument here, though: it was specifically tuned to penalize evasiveness, a methodological choice that reshapes what the resulting numbers can say about the tension.
RL-CAI (RLAIF-trained Constitutional AI model) (3 connections)
- RL-CAI (RLAIF-trained Constitutional AI model) — eliminates → Evasiveness
- RL-CAI (RLAIF-trained Constitutional AI model) — produces → Pareto Improvement (helpfulness vs. harmlessness)
- SL-CAI (Supervised-Learning Constitutional AI model) — seeds the policy and comparison data for → RL-CAI (RLAIF-trained Constitutional AI model)
RL-CAI is measured on that same Elo scale against both baselines, and its training signal never rewards refusal for its own sake the way HH RLHF's did, so it eliminates evasiveness rather than trading it back in for harmlessness.
Pareto Improvement (helpfulness vs. harmlessness) (1 connection)
- RL-CAI (RLAIF-trained Constitutional AI model) — produces → Pareto Improvement (helpfulness vs. harmlessness)
Because RL-CAI achieves that without sliding back along the helpful-model/HH-model line, it produces the paper's headline finding, pareto improvement: Constitutional RL shifts the entire helpfulness-harmlessness frontier outward rather than merely moving along the standard RLHF tradeoff curve this walk began by documenting.

Elo as the measuring stick concept (4 connections)

Crowdworker pairwise preferences over model snapshots become Elo ratings for helpfulness and harmlessness separately — and the harmlessness instructions were deliberately tuned so that evasion loses points (constitutional-ai, §"3.3 Main Results", p. 8).

connections

Absolute Harmfulness Score — cross checks → Elo Score (helpfulness/harmlessness)
Elo Score (helpfulness/harmlessness) — supplies the explanatory analogy for → Bradley-Terry Model
Elo Score (helpfulness/harmlessness) — was tuned to penalize → Evasiveness
Elo Score (helpfulness/harmlessness) — generalizes → Win rate (preference against baseline) hindsight

The frontier claim concept (1 connection)

The paper's strongest sentence: constitutional RL is both more helpful and more harmless than its RLHF baseline — an outward shift of the frontier rather than movement along it (constitutional-ai, §"1.1 Motivations", p. 3).

connections

RL-CAI (RLAIF-trained Constitutional AI model) — produces → Pareto Improvement (helpfulness vs. harmlessness)

A second, absolute instrument concept (1 connection)

A 0-4 harmfulness scale predicted by a separately finetuned model gives the relative Elo picture an absolute cross-check; the two instruments broadly agree, which is the point of running both (constitutional-ai, §"4.5 Absolute Harmfulness Score", p. 14).

connections

Absolute Harmfulness Score — cross checks → Elo Score (helpfulness/harmlessness)

Goodhart, again concept (4 connections)

Over-train against the preference model and RL-CAI grows harsh and accusatory toward harmless-seeming prompts: the 2016 law showing up empirically inside the 2022 pipeline (constitutional-ai, §"4.3 Main Results", p. 12).

connections

Goodhart's law — is instantiated through mislabeled proxy in → Evasiveness hindsight
Goodhart's law — operates at the subfeature level in → Excessive hedging on simple questions hindsight
Goodhart's law — is empirically instantiated by → Reward model over-optimization hindsight
Sycophancy — instantiates → Goodhart's law hindsight

What cheap labels unlock next concept (4 connections)

Named as future work: with AI-generated labels, continually refitting the preference model on the current policy's outputs becomes automatable rather than a crowdsourcing project (constitutional-ai, §"6.1 Future Directions", p. 15).

connections

Label Annealing — paces the query rate within → Iterated (Online) Training
Iterated (Online) Training — is the cause of → Non-Stationary Reward Function Challenge
Iterated (Online) Training — guards against → Reward hacking (avoiding)
RL from AI Feedback (RLAIF) — makes newly automatable → Iterated (Online) Training

The pipeline again, with the human half swapped out: harmlessness labels now come from the model itself, steered by a written constitution. Nearly the whole introduced shelf is the machinery of that swap — the constitution and its two principle sets, critique-and-revision, the feedback model, RLAIF and its two trained endpoints. The refined shelf is small but is the paper's thesis: scalable oversight, an abstract problem since 2016, acquires an operational meaning — use AI supervision where human supervision cannot scale — and the harmlessness evaluations are sharpened to catch the failure mode the paper names, Evasiveness, harmlessness achieved by refusing to engage. The inherited shelf is effectively InstructGPT's pipeline plus the 2016 failure vocabulary: Goodhart's law returns here empirically, as over-training against the preference model.

Read at

Introduced here introduced (19 concepts, in reading order)

Constitutional AI (CAI) (1 connection) §“Constitutional AI: Harmlessness from AI Feedback”, p. 1
- Constitution (set of principles) — replaces human harm labels in → Constitutional AI (CAI)
A two-stage method for training a harmless language model assistant using only a set of written principles rather than human harmfulness labels, combining supervised self-critique/revision with reinforcement learning from AI-generated preference labels.
RL from AI Feedback (RLAIF) (4 connections) §“Abstract”, p. 1
- RL from AI Feedback (RLAIF) — delegates harmlessness comparisons to → Feedback Model
- RL from AI Feedback (RLAIF) — makes newly automatable → Iterated (Online) Training
- Open-ended generation evaluation with GPT-4 rating — shares the ai judge move with → RL from AI Feedback (RLAIF) hindsight
- Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)
A variant of RLHF in which the harmlessness preference labels used to train the reward/preference model are generated by an AI feedback model instead of human crowdworkers.
Pareto Improvement (helpfulness vs. harmlessness) (1 connection) §“1.1 Motivations”, p. 3
- RL-CAI (RLAIF-trained Constitutional AI model) — produces → Pareto Improvement (helpfulness vs. harmlessness)
The finding that Constitutional RL shifts the helpfulness-harmlessness tradeoff frontier outward rather than just moving along the standard RLHF tradeoff curve.
Evasiveness (7 connections) §“A Harmless but Non-Evasive (Still Helpful) Assistant”, p. 4
- Elo Score (helpfulness/harmlessness) — was tuned to penalize → Evasiveness
- Excessive hedging on simple questions — is the harmlessness axis mirror of → Evasiveness hindsight
- Goodhart's law — is instantiated through mislabeled proxy in → Evasiveness hindsight
- Helpfulness-Harmlessness Tension — collapses into → Evasiveness
- Per-token KL penalty from the SFT model — was not the fix applied to → Evasiveness hindsight
- Refusal (as a steerable behavior) — generalizes → Evasiveness hindsight
- RL-CAI (RLAIF-trained Constitutional AI model) — eliminates → Evasiveness
A named failure mode in which a harmlessness-trained assistant gives canned refusals (e.g. "I can't answer that") to sensitive queries instead of engaging, sacrificing transparency and helpfulness for apparent safety.
Constitution (set of principles) (1 connection) §“1.2 The Constitutional AI Approach”, p. 5
- Constitution (set of principles) — replaces human harm labels in → Constitutional AI (CAI)
The small set of natural-language principles/instructions that provide the sole source of human guidance for steering model critiques, revisions, and preference comparisons in CAI.
Hybrid Human/AI Preference Model (1 connection) §“1.2 The Constitutional AI Approach”, p. 5
- Feedback Model — supplies the harmlessness half of → Hybrid Human/AI Preference Model
A preference model trained on a mixture of human-labeled helpfulness comparisons and AI-labeled harmlessness comparisons.
Critique and Revision (3 connections) §“3 Constitutional AI: Critiques, Revisions, and Supervised Learning”, p. 7
- Direct Revision (critique-free ablation) — tests the necessity of critique in → Critique and Revision
- Few-shot Prompts for SL-CAI (Appendix E.1) — prevents role confusion in → Critique and Revision
- Critique and Revision — supplies the finetuning data for → SL-CAI (Supervised-Learning Constitutional AI model)
The supervised-stage pipeline in which a model critiques its own harmful response according to a sampled constitutional principle and then revises the response to remove the identified harm.
SL-CAI (Supervised-Learning Constitutional AI model) (2 connections) §“3.2 Datasets and Training”, p. 8
- Critique and Revision — supplies the finetuning data for → SL-CAI (Supervised-Learning Constitutional AI model)
- SL-CAI (Supervised-Learning Constitutional AI model) — seeds the policy and comparison data for → RL-CAI (RLAIF-trained Constitutional AI model)
The model produced by finetuning a pretrained LM on constitutionally revised harmlessness responses plus helpfulness samples, serving as the initial policy for the RL stage.
Direct Revision (critique-free ablation) (1 connection) §“3.5 Are Critiques Necessary?”, p. 10
- Direct Revision (critique-free ablation) — tests the necessity of critique in → Critique and Revision
An ablation of the SL-CAI pipeline that skips the critique step and asks the model to generate a revision directly, compared against critiqued revisions to test whether the critique step is necessary.
Feedback Model (4 connections) §“4.1 Method”, p. 10
- Feedback Model — supplies the harmlessness half of → Hybrid Human/AI Preference Model
- Multiple-Choice Comparison Format — turns into a preference labeler → Feedback Model
- Principle Ensembling — makes more robust → Feedback Model
- RL from AI Feedback (RLAIF) — delegates harmlessness comparisons to → Feedback Model
An independent (typically pretrained) language model presented with a harmful prompt and a pair of responses that produces the multiple-choice harmlessness preference label used to construct the AI feedback dataset.
Multiple-Choice Comparison Format (3 connections) §“4.1 Method”, p. 10
- Multiple-Choice Comparison Format — turns into a preference labeler → Feedback Model
- Model Calibration (well-calibrated multiple-choice probabilities) — is the precondition for → Multiple-Choice Comparison Format
- Multiple-choice contrast-pair format — repurposes the ab mechanic of → Multiple-Choice Comparison Format hindsight
The prompt template that presents a conversation, a constitutional principle, and two candidate responses labeled (A)/(B), from which the feedback model's answer log-probabilities become the preference label.
RL-CAI (RLAIF-trained Constitutional AI model) (3 connections) §“4 Constitutional AI: Reinforcement Learning from AI Feedback”, p. 10
- RL-CAI (RLAIF-trained Constitutional AI model) — eliminates → Evasiveness
- RL-CAI (RLAIF-trained Constitutional AI model) — produces → Pareto Improvement (helpfulness vs. harmlessness)
- SL-CAI (Supervised-Learning Constitutional AI model) — seeds the policy and comparison data for → RL-CAI (RLAIF-trained Constitutional AI model)
The final policy produced by running RL against a preference model trained partly on AI-generated harmlessness labels, starting from the SL-CAI snapshot.
Principle Ensembling (2 connections) §“4.1 Method”, p. 11
- Principle Ensembling — makes more robust → Feedback Model
- Principle Ensembling — diversifies use of → Principles for SL-CAI (Appendix C.1)
Randomly sampling a different constitutional principle for each comparison label (rather than reusing one principle for all labels) to produce more robust preference-model behavior.
Preference Label Clamping (Soft vs. Hard vs. Clamped) (1 connection) §“4.3 Main Results”, p. 13
- Chain-of-Thought (CoT) Prompting — creates the overconfidence that necessitates → Preference Label Clamping (Soft vs. Hard vs. Clamped)
A technique for constructing PM training targets from feedback-model outputs, comparing raw soft (normalized log-probability) labels, hard 0/1 labels, and probabilities clamped to a fixed range (e.g. 40-60%) to counteract CoT-induced overconfidence.
Harmful Behavior Identification/Classification Evaluations (Appendix B) (1 connection) §“B Identifying and Classifying Harmful Conversations”, p. 19
- Red Teaming — supplies the transcripts for → Harmful Behavior Identification/Classification Evaluations (Appendix B)
Two multiple-choice evaluations built from red-teaming transcripts: identifying harmful vs. ethical assistant behavior, and classifying harmful conversations into one of nine categories, used to test whether models can replicate crowdworker harm judgments.
Principles for SL-CAI (Appendix C.1) (2 connections) §“C.1 Principles for SL-CAI”, p. 20
- Principle Ensembling — diversifies use of → Principles for SL-CAI (Appendix C.1)
- Principles for SL-CAI (Appendix C.1) — is mirrored in a different shape by → Principles for RL-CAI (Appendix C.2)
The specific list of 16 critique/revision instruction pairs used to steer the supervised critique-and-revision phase.
Principles for RL-CAI (Appendix C.2) (1 connection) §“C.2 Principles for RL-CAI”, p. 22
- Principles for SL-CAI (Appendix C.1) — is mirrored in a different shape by → Principles for RL-CAI (Appendix C.2)
The specific list of 16 multiple-choice comparison principles used by the feedback model to label which of two responses is more harmless.
Few-shot Prompts for SL-CAI (Appendix E.1) (1 connection) §“E.1 Few-shot Prompts for SL-CAI”, p. 29
- Few-shot Prompts for SL-CAI (Appendix E.1) — prevents role confusion in → Critique and Revision
Hand-written few-shot conversation/critique/revision examples prepended to the SL-CAI critique- and revision-generating model to prevent point-of-view confusion between critique and revision roles.
Chain-of-Thought Prompts for RL-CAI (Appendix E.2) (1 connection) §“E.2 Chain-of-Thought Prompts for RL-CAI”, p. 31
- "Let's think step-by-step" zero-shot CoT trigger — is paired with → Chain-of-Thought Prompts for RL-CAI (Appendix E.2)
Hand-written few-shot examples, each with a substitutable principle placeholder, used to prompt the feedback model to produce step-by-step reasoning before its multiple-choice harmlessness judgment.

Refined here refined (3 concepts, in reading order)

Scalable oversight (4 connections) §“Scaling Supervision”, p. 2
- Scalable oversight — is addressed by adapting → Distant supervision
- Scalable oversight — is addressed by → Hierarchical reinforcement learning (HRL)
- Scalable oversight — is retroactively operationalized by → Reward model (RM) hindsight
- Scalable oversight — is operationalized as → Semi-supervised reinforcement learning
An umbrella term for techniques that use AI systems to help humans supervise other AIs more efficiently, of which Constitutional AI is presented as an extreme instance.
HHH Evaluation Set (binary comparisons) (3 connections) §“2 Evaluating the Potential for AI Supervision of HHH”, p. 6
- HHH Evaluation Set (binary comparisons) — is hosted as a subset of → BIG-Bench
- Helpful, honest, harmless (HHH) alignment framework — is operationalized as → HHH Evaluation Set (binary comparisons)
- Model Calibration (well-calibrated multiple-choice probabilities) — is empirically verified on → HHH Evaluation Set (binary comparisons)
A benchmark of binary response-comparison questions testing whether a model can identify the more helpful, honest, and harmless reply; originally 221 comparisons from Askell et al. (2021) on BIG-Bench, extended here with 217 additional harder harmlessness-focused comparisons.
Absolute Harmfulness Score (1 connection) §“4.5 Absolute Harmfulness Score”, p. 14
- Absolute Harmfulness Score — cross checks → Elo Score (helpfulness/harmlessness)
A single-conversation harmfulness metric (0-4 scale) predicted by a model finetuned with L2 loss on crowdworker absolute-harm ratings, used as an additional evaluation signal alongside pairwise Elo comparisons.

Inherited — used, not invented here inherited (20 concepts, in reading order)

Helpful, honest, harmless (HHH) alignment framework (2 connections) §“1 Introduction”, p. 2
- Evil (persona trait) — inverts the baseline encoded by → Helpful, honest, harmless (HHH) alignment framework hindsight
- Helpful, honest, harmless (HHH) alignment framework — is operationalized as → HHH Evaluation Set (binary comparisons)
The three-part objective for AI assistant behavior established in prior work, which CAI aims to satisfy without human harmfulness labels.
Helpfulness-Harmlessness Tension (2 connections) §“1.1 Motivations”, p. 2
- Helpfulness-Harmlessness Tension — collapses into → Evasiveness
- Refusal (as a steerable behavior) — quantifies → Helpfulness-Harmlessness Tension hindsight
The observed tradeoff whereby increasing a model's willingness to help tends to increase its potential for harm, and increasing harmlessness training tends to make a model more evasive and less helpful.
Reinforcement learning from human feedback (RLHF) (6 connections) §“1 Introduction”, p. 2
- Asynchronous Reward-Learning Architecture — gives concrete shape to → Reinforcement learning from human feedback (RLHF)
- Base vs. RLHF-chat model representation comparison — measures the effect of → Reinforcement learning from human feedback (RLHF) hindsight
- Contrastive Activation Addition (CAA) — operates on top of → Reinforcement learning from human feedback (RLHF) hindsight
- Reward model (RM) — anchors the reward signal of → Reinforcement learning from human feedback (RLHF)
- Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)
- Sycophancy — misgeneralizes the training objective of → Reinforcement learning from human feedback (RLHF) hindsight
The established technique of training a policy with RL against a preference model fit to human comparison judgments; CAI extends and partially replaces it for harmlessness.
Automated Red Teaming (2 connections) §“A Harmless but Non-Evasive (Still Helpful) Assistant”, p. 4
- CAA as an adversarial red-teaming tool (proposed application) — skips the prompt search of → Automated Red Teaming hindsight
- Red Teaming — is the manual process that scales into → Automated Red Teaming
A related approach (Perez et al., 2022) of using language models themselves to generate red-team prompts, noted as a future direction that CAI's non-evasiveness should make easier to scale up.
Reward model (RM) (11 connections) §“1.2 The Constitutional AI Approach”, p. 5
- Adaptive L2 Regularization for the Reward Predictor — bounds the generalization gap of → Reward model (RM)
- Asynchronous Reward-Learning Architecture — trades its concurrency for a fixed dataset in → Reward model (RM) hindsight
- Bradley-Terry Model — supplies the loss for → Reward model (RM)
- Reward model (RM) — is overbuilt in anticipation of → Novel Behavior Training
- Reward model (RM) — supervises from below → PPO fine-tuning stage (RLHF policy optimization)
- Recurrent Reward Model (Proposed Extension) — would generalize the fixed window of → Reward model (RM)
- Reward model (RM) — anchors the reward signal of → Reinforcement learning from human feedback (RLHF)
- Reward model pairwise comparison loss — under determines the scale of → Reward model (RM)
- Reward Prediction Normalization — fixes the underdetermined scale of → Reward model (RM)
- Scalable oversight — is retroactively operationalized by → Reward model (RM) hindsight
- Reward model (RM) — back selects the checkpoint of → Supervised fine-tuned (SFT) model
A model trained on comparison data to assign a scalar score to a response, used as the reward signal for RL, following the RLHF pipeline of prior work.
Helpful RLHF Model (1 connection) §“1.4 Models and Data”, p. 6
- Helpful RLHF Model — traces the tradeoff curve with → HH RLHF Model
An RLHF policy trained using only helpfulness human-feedback comparisons, used as the harmful-response-generating starting point for the SL-CAI critique/revision pipeline.
HH RLHF Model (1 connection) §“1.4 Models and Data”, p. 6
- Helpful RLHF Model — traces the tradeoff curve with → HH RLHF Model
An RLHF policy trained on both helpfulness and harmlessness human-feedback comparisons, used as a baseline comparison against RL-CAI throughout the paper.
Red Teaming (5 connections) §“3.1 Method”, p. 7
- Red Teaming — is the manual process that scales into → Automated Red Teaming
- CAA as an adversarial red-teaming tool (proposed application) — relocates the attack surface of → Red Teaming hindsight
- Red Teaming — supplies the transcripts for → Harmful Behavior Identification/Classification Evaluations (Appendix B)
- Jailbreaks — is the target phenomenon of → Red Teaming
- PALMS (Process for Adapting Language Models to Society) — provides an independent check outside → Red Teaming
The practice of having crowdworkers conduct adversarial conversations designed to bait a language model into harmful outputs, used as the source of harmful prompts for both SL-CAI and RL-CAI training.
Elo Score (helpfulness/harmlessness) (4 connections) §“3.3 Main Results”, p. 8
- Absolute Harmfulness Score — cross checks → Elo Score (helpfulness/harmlessness)
- Elo Score (helpfulness/harmlessness) — supplies the explanatory analogy for → Bradley-Terry Model
- Elo Score (helpfulness/harmlessness) — was tuned to penalize → Evasiveness
- Elo Score (helpfulness/harmlessness) — generalizes → Win rate (preference against baseline) hindsight
A rating computed from crowdworker pairwise model-comparison preferences, used throughout the paper to track relative helpfulness and harmlessness of model snapshots during training.
Model Calibration (well-calibrated multiple-choice probabilities) (2 connections) §“4.1 Method”, p. 10
- Model Calibration (well-calibrated multiple-choice probabilities) — is empirically verified on → HHH Evaluation Set (binary comparisons)
- Model Calibration (well-calibrated multiple-choice probabilities) — is the precondition for → Multiple-Choice Comparison Format
The property, established in prior work, that language models' probabilities on multiple-choice questions are reasonably well-calibrated, which CAI relies on to treat feedback-model output probabilities as usable soft preference labels.
"Let's think step-by-step" zero-shot CoT trigger (1 connection) §“Chain-of-Thought Prompting”, p. 11
- "Let's think step-by-step" zero-shot CoT trigger — is paired with → Chain-of-Thought Prompts for RL-CAI (Appendix E.2)
A specific zero-shot prompt phrase from Kojima et al. (2022) appended to elicit chain-of-thought reasoning from the feedback model before it commits to a multiple-choice harmlessness judgment.
Chain-of-Thought (CoT) Prompting (1 connection) §“Chain-of-Thought Prompting”, p. 11
- Chain-of-Thought (CoT) Prompting — creates the overconfidence that necessitates → Preference Label Clamping (Soft vs. Hard vs. Clamped)
A prompting technique that elicits intermediate step-by-step reasoning from a language model before its final answer, used here both for SL critiques and to make the RL feedback model's harmlessness judgments more accurate and transparent.
Context Distillation §“4.2 Datasets and Training”, p. 11
A prior finetuning technique of distilling a lengthy prompt's behavior into model weights, used for RLHF models in earlier work but not for the RLHF models trained directly in this paper.
Goodhart's law (4 connections) §“4.3 Main Results”, p. 12
- Goodhart's law — is instantiated through mislabeled proxy in → Evasiveness hindsight
- Goodhart's law — operates at the subfeature level in → Excessive hedging on simple questions hindsight
- Goodhart's law — is empirically instantiated by → Reward model over-optimization hindsight
- Sycophancy — instantiates → Goodhart's law hindsight
A named failure mode in which over-training against the preference-model reward causes RL-CAI to become overly harsh, boilerplate, or accusatory toward harmless-seeming prompts, degrading true response quality despite improving proxy reward.
InstructGPT (1 connection) §“5 Related Work”, p. 14
- InstructGPT — outperforms at 100x smaller scale → GPT-3 (pretrained language model)
OpenAI's instruction-tuned language model (Ouyang et al., 2022), cited as related RLHF-based alignment work and as a source of comparison prompts in Appendix D.
LaMDA (1 connection) §“5 Related Work”, p. 14
- LaMDA — plays a different comparative role than → Sparrow
Google's dialogue language model (Thoppilan et al., 2022), cited as related aligned-dialogue-agent work and as a source of comparison prompts and reference responses in Appendix D.
Sparrow (1 connection) §“5 Related Work”, p. 14
- LaMDA — plays a different comparative role than → Sparrow
DeepMind's dialogue agent (Glaese et al., 2022), cited as related work whose decomposition of harmlessness into targeted rules parallels this paper's use of constitutional principles.
Iterated (Online) Training (4 connections) §“6.1 Future Directions”, p. 15
- Label Annealing — paces the query rate within → Iterated (Online) Training
- Iterated (Online) Training — is the cause of → Non-Stationary Reward Function Challenge
- Iterated (Online) Training — guards against → Reward hacking (avoiding)
- RL from AI Feedback (RLAIF) — makes newly automatable → Iterated (Online) Training
A training regime from prior work in which the preference model is periodically refreshed with new comparison data collected on the current policy's outputs, keeping the PM on-distribution; noted as a future direction now automatable with AI feedback.
BIG-Bench (1 connection) §“B Identifying and Classifying Harmful Conversations”, p. 20
- HHH Evaluation Set (binary comparisons) — is hosted as a subset of → BIG-Bench
A large collaborative benchmark suite for language models that hosts the original HHH evaluation set used as a comparison baseline.
PALMS (Process for Adapting Language Models to Society) (1 connection) §“D.1 PALMS Sensitive Questions”, p. 23
- PALMS (Process for Adapting Language Models to Society) — provides an independent check outside → Red Teaming
A prior values-targeted dataset of sensitive questions (Solaiman & Dennison, 2021) used here as a fixed prompt set for qualitatively comparing HH RLHF and RL-CAI responses.

.tab-panel[hidden], .story-panel[hidden] { display: block; } .tabs, .subtabs, .tree-controls { display: none; } .panel-title { display: block; } .story-panel + .story-panel { margin-top: 2.5rem; border-top: 1px solid var(--stroke-strong); padding-top: 1.5rem; }