The helpfulness-harmlessness tradeoff and CAI's claim to beat it

Members — a guided walk

1. Helpfulness-Harmlessness Tension

   Constitutional AI's central empirical argument concerns a documented tradeoff: increasing a model's willingness to help tends to increase its potential for harm, while increasing harmlessness training tends to make a model more evasive rather than genuinely safer.

2. Evasiveness

   That tension collapses concretely into evasiveness: when feedback rewards harmlessness without also penalizing refusal, the cheapest way for a model to score well on harmful prompts is to disengage with a canned "I can't answer that" rather than think the request through, exactly what Bai et al.'s prior HH RLHF training exhibited.

3. Refusal (as a steerable behavior)

   Refusal gives the same tension a direct dial, one Steering Llama 2 via Contrastive Activation Addition (2023) turns rather than just observes: a custom contrastive dataset spans both sensitive and entirely benign questions, broader than evasiveness's narrower focus on canned refusals to sensitive queries specifically. Moved up or down with a signed multiplier, refusal turns the helpful-versus-harmless tension from a property observed across differently trained models, like the two RLHF baselines below, into a continuous quantity measurable within a single model.

4. Helpful RLHF Model

   The helpful RLHF model is one of two baselines the paper uses to trace that tradeoff empirically: a policy trained on helpfulness comparisons only, serving as the harmful-response-generating starting point for the SL-CAI critique-and-revision pipeline.

5. HH RLHF Model

   The HH RLHF model differs from it in exactly one respect, harmlessness comparisons added to the same pipeline, and that single difference is what traces the tradeoff curve empirically: the helpful model is more helpful but more harmful, the HH model less harmful but more evasive, plotting two ends of the same line.

6. Elo Score (helpfulness/harmlessness)

   The Elo score is the instrument used to plot that line, a rating computed from crowdworker pairwise comparisons tracking relative helpfulness and harmlessness across training snapshots. It is not a neutral instrument here, though: it was specifically tuned to penalize evasiveness, a methodological choice that reshapes what the resulting numbers can say about the tension.

7. RL-CAI (RLAIF-trained Constitutional AI model)

   RL-CAI is measured on that same Elo scale against both baselines, and its training signal never rewards refusal for its own sake the way HH RLHF's did, so it eliminates evasiveness rather than trading it back in for harmlessness.

8. Pareto Improvement (helpfulness vs. harmlessness)

   Because RL-CAI achieves that without sliding back along the helpful-model/HH-model line, it produces the paper's headline finding, pareto improvement: Constitutional RL shifts the entire helpfulness-harmlessness frontier outward rather than merely moving along the standard RLHF tradeoff curve this walk began by documenting.

Superthemes

• Defining the target before optimizing for it: instruction-following, HHH, and the tradeoff it creates

Edges explored within this theme

• Helpfulness-Harmlessness Tension — collapses into → Evasiveness
• Helpful RLHF Model — traces the tradeoff curve with → HH RLHF Model
• RL-CAI (RLAIF-trained Constitutional AI model) — eliminates → Evasiveness
• RL-CAI (RLAIF-trained Constitutional AI model) — produces → Pareto Improvement (helpfulness vs. harmlessness)
• Elo Score (helpfulness/harmlessness) — was tuned to penalize → Evasiveness
• Refusal (as a steerable behavior) — quantifies → Helpfulness-Harmlessness Tension hindsight

Super edges touching this theme

• Helpful, honest, harmless: a shared definition of alignment — narrows to two of its three terms in → The helpfulness-harmlessness tradeoff and CAI's claim to beat it
• Automatic overlap scores versus human and preference judgments — supplies the frontier tracking instrument for → The helpfulness-harmlessness tradeoff and CAI's claim to beat it hindsight

Sources

Every passage behind this theme’s member concepts.