Model Calibration (well-calibrated multiple-choice probabilities)
Constitutional AI: Harmlessness from AI Feedback — inherited
The property, established in prior work, that language models' probabilities on multiple-choice questions are reasonably well-calibrated, which CAI relies on to treat feedback-model output probabilities as usable soft preference labels.
A model is well calibrated if, whenever it claims eighty percent confidence, it turns out to be right about eighty percent of the time. That property, established well before this paper, is what lets Constitutional AI treat a model's raw output probabilities as a meaningful signal rather than noise.
What calibration means is explained first, then why it is a load-bearing assumption behind the paper's soft preference labels, before connecting to the HHH evaluation set and the multiple-choice format that both rely on it.
What calibration means
Calibration is a general property of probabilistic predictors that predates Constitutional AI. A model is well-calibrated if, among all the times it outputs a given confidence level $p$ for its prediction, the predictions are correct a fraction $p$ of the time: $$P\big(Y = \hat{Y} \mid \hat{P}(\hat{Y}) = p\big) = p \quad \text{for all } p \in [0,1].$$ A model can be highly accurate yet poorly calibrated, systematically over- or under-confident, so calibration is checked separately from accuracy, typically by binning predictions by confidence and comparing each bin's average confidence to its empirical accuracy, summarized in a single number as expected calibration error, $\text{ECE} = \sum_m \frac{|B_m|}{n}\,|\text{acc}(B_m) - \text{conf}(B_m)|$. For language models specifically, Kadavath et al. (2022) established that a model's next-token probabilities on multiple-choice questions are reasonably well-calibrated in this sense, a finding Constitutional AI relies on directly.
The load-bearing assumption behind soft preference labels
Constitutional AI invokes calibration to justify a specific design choice: treating the feedback model's raw output probabilities on a multiple-choice comparison as usable soft preference-label targets rather than noise, since such targets "will be fairly well-calibrated...since they are multiple choice responses" (constitutional-ai, §"4.1 Method", p. 10) (Model Calibration (well-calibrated multiple-choice probabilities) — is the precondition for → Multiple-Choice Comparison Format). Without this property holding, there would be no principled reason to prefer soft probability targets over hard 0/1 labels. The paper does not leave this an unverified assumption: it is empirically checked on the HHH Evaluation Set (binary comparisons), where "the feedback model's log-probabilities are reasonably well-calibrated" (constitutional-ai, §"4.3 Main Results", p. 12) (Model Calibration (well-calibrated multiple-choice probabilities) — is empirically verified on → HHH Evaluation Set (binary comparisons)), closing the loop between premise and evidence. Both edges sit inside the Helpful, honest, harmless: a shared definition of alignment and Engineering a language model into a usable preference labeler themes and the Engineering the exact prompt or format needed to manufacture a model behavior connective theme, which traces the chain of narrow prompting and calibration choices the feedback-model pipeline depends on end to end.