RL from AI Feedback (RLAIF) — delegates harmlessness comparisons to → Feedback Model

RLAIF's substitution of an AI labeler for a human one is carried out entirely by the feedback model, which is shown the harmful prompt and a pair of responses formatted as an explicit multiple-choice question ('(A) [Response A] (B) [Response B] The answer is:') and scored by the log-probability the model assigns to each option (constitutional-ai, §"4.1 Method", p. 10). Using a probability rather than a forced binary choice is deliberate: soft labels are expected to be reasonably well-calibrated for multiple-choice-formatted questions, so the raw probabilities can double as usable preference-model training targets rather than noise. The chain-of-thought variant of this same feedback model actually broke that assumption — its probabilities collapsed toward 0 or 1 — and had to be manually clamped to a 40-60% range to keep training stable, showing the calibration premise is a real constraint on the method, not just a convenience.