Helpful RLHF Model
Constitutional AI: Harmlessness from AI Feedback — inherited
An RLHF policy trained using only helpfulness human-feedback comparisons, used as the harmful-response-generating starting point for the SL-CAI critique/revision pipeline.
Not every model discussed here was trained for this paper. This one is an earlier RLHF policy, shaped purely by human judgments of helpfulness with no harmlessness training at all, and Constitutional AI borrows it simply as a convenient source of responses that still need fixing.
The page explains where this policy came from and the specific jobs it plays here, then connects to the harmlessness-trained RLHF model it is compared against.
An RLHF policy, predating this paper
Reinforcement learning from human feedback (RLHF) trains a policy in three parts: a base language model is finetuned on demonstrations, a reward model is fit to human pairwise comparisons of that policy's outputs, and the policy is then optimized against the reward model with reinforcement learning, typically regularized by a KL penalty against the starting policy so it does not drift into degenerate outputs that exploit the reward model. Given two outputs $y_1, y_2$ for the same input $x$, a reward model $r_\theta$ is fit so that the probability a human prefers $y_1$ follows a Bradley-Terry form, $$ P(y_1 \succ y_2 \mid x) = \frac{\exp(r_\theta(x,y_1))}{\exp(r_\theta(x,y_1)) + \exp(r_\theta(x,y_2))}, $$ and the policy $\pi_\phi$ is trained to maximize $E[\,r_\theta(x,y) - \beta \log(\pi_\phi(y\mid x)/\pi_{\text{ref}}(y\mid x))\,]$. This technique predates all three papers in this corpus in its modern form (Christiano et al. 2017) and is refined by InstructGPT for open-ended instruction-following (Reinforcement learning from human feedback (RLHF); instructgpt, §"3.1 High-level methodology", p. 6); Constitutional AI inherits it directly from Anthropic's own prior work (Bai et al. 2022) rather than introducing it.
What makes it helpful specifically, and its two jobs here
The helpful RLHF model is the instance of this procedure where the reward model was fit exclusively to helpfulness comparisons, with no harmfulness data included (constitutional-ai, §"1.4 Models and Data", p. 6). Within Constitutional AI it plays two roles. First, it is the response-generating starting point for Critique and Revision: because it was never trained to avoid harm, prompting it with adversarial red-team inputs reliably produces the harmful responses the critique-and-revision pipeline needs to revise. Second, it is one endpoint of the standard-RLHF reference line plotted against RL-CAI (RLAIF-trained Constitutional AI model)'s results: the paper reports "the helpful RLHF model is more helpful but also more harmful than HH RLHF" (constitutional-ai, §"3.3 Main Results", p. 8), and this single fixed difference between it and HH RLHF Model is what traces the tradeoff curve CAI's Pareto Improvement (helpfulness vs. harmlessness) is measured against (Helpful RLHF Model — traces the tradeoff curve with → HH RLHF Model).