HH RLHF Model
Constitutional AI: Harmlessness from AI Feedback — inherited
An RLHF policy trained on both helpfulness and harmlessness human-feedback comparisons, used as a baseline comparison against RL-CAI throughout the paper.
This is the model Constitutional AI actually has to beat. An earlier RLHF policy trained on comparisons covering both helpfulness and harmlessness together, it sets the bar the paper's own method is measured against throughout.
Where this baseline came from, and why its shortcomings motivate the paper's redesign, fill most of the page, before connecting to the helpfulness-only model it complements.
An RLHF policy trained on two kinds of comparisons, predating this paper
The HH RLHF model is produced by the same Reinforcement learning from human feedback (RLHF) procedure as Helpful RLHF Model — reward model fit to human pairwise comparisons, policy optimized against it under a KL penalty — but with one difference: its reward model is fit to comparisons covering both helpfulness and harmlessness, collected separately by having crowdworkers red-team the model for the latter (constitutional-ai, §"1.4 Models and Data", p. 6). Like helpful-rlhf-model, this specific policy predates Constitutional AI: it is inherited from Anthropic's prior paper on training a helpful and harmless assistant with human feedback (Bai et al., 2022), which Constitutional AI cites throughout as the source of both its training procedure and its documented failure mode.
A baseline that motivates the whole paper's redesign
HH RLHF is the second endpoint of the standard-RLHF tradeoff line, more harmless but less helpful than helpful-rlhf-model (Helpful RLHF Model — traces the tradeoff curve with → HH RLHF Model), and Constitutional AI treats its known behavior as the problem to be solved. Prior work found HH RLHF "often evasive when presented with sensitive discussions, giving canned responses like 'I can't answer that'" (constitutional-ai, §"4.4 Harmlessness vs. Evasiveness", p. 13), a consequence of crowdworkers having been asked to prefer the more harmless response without being told to penalize refusal. This is the specific failure that motivates RL-CAI (RLAIF-trained Constitutional AI model)'s redesigned Principles for RL-CAI (Appendix C.2) and Elo-scoring instructions, and the paper notes that its own re-measurement narrows the harmlessness gap between HH RLHF and helpful-rlhf-model relative to the earlier paper, because crowdworkers here were separately instructed to penalize evasiveness — a methodological choice, not a change in either model, that compresses HH RLHF's apparent advantage (constitutional-ai, §"3.3 Main Results", footnote 8, p. 8).