Multiple-Choice Comparison Format
Constitutional AI: Harmlessness from AI Feedback — introduced
The prompt template that presents a conversation, a constitutional principle, and two candidate responses labeled (A)/(B), from which the feedback model's answer log-probabilities become the preference label.
Turning next-token probabilities into an actual preference takes some engineering. This format lays out a conversation, a principle, and two labeled responses, then reads off which label the model finds more probable as its answer.
How the trick turns ordinary next-token prediction into a usable preference signal is explained first, along with the assumption it quietly leans on, before connecting to the feedback model that uses it and the calibration property it depends on.
Manufacturing a preference signal from next-token prediction
The multiple-choice comparison format is the specific prompt template that lets an ordinary language model produce a usable preference judgment. It presents a conversation, a constitutional principle, and two labeled candidate responses, then asks the model to complete "The answer is:" (constitutional-ai, §"4.1 Method", p. 10). Reading off the model's log-probability for each option token converts unstructured next-token prediction into a scalar comparison: writing $\ell_A$ and $\ell_B$ for the model's log-probabilities on "(A)" and "(B)", the normalized soft label used as a preference-model target is $$ p_A = \frac{\exp(\ell_A)}{\exp(\ell_A) + \exp(\ell_B)}. $$ Without this templating there is no way to extract a comparable preference signal from a model that was never trained to produce one (Multiple-Choice Comparison Format — turns into a preference labeler → Feedback Model) — it is what turns a plain pretrained model into the Feedback Model.
Resting on calibration
This move is only trustworthy because of Model Calibration (well-calibrated multiple-choice probabilities): prior work established that language-model probabilities on multiple-choice questions are reasonably well-calibrated estimates rather than arbitrary numbers, which is what justifies using the normalized probability $p_A$ directly as a soft training target instead of forcing a hard 0/1 label (Model Calibration (well-calibrated multiple-choice probabilities) — is the precondition for → Multiple-Choice Comparison Format; constitutional-ai, §"4.1 Method", p. 10). As a child of RL from AI Feedback (RLAIF), the format is the piece of machinery that sits between an off-the-shelf language model and a working preference labeler, and it belongs to the Engineering the exact prompt or format needed to manufacture a model behavior connective theme alongside the other narrow prompt-engineering decisions the feedback pipeline depends on.