Feedback Model
Constitutional AI: Harmlessness from AI Feedback — introduced
An independent (typically pretrained) language model presented with a harmful prompt and a pair of responses that produces the multiple-choice harmlessness preference label used to construct the AI feedback dataset.
There is nothing special about the feedback model's architecture; it is an ordinary pretrained language model, just handed a new job. Shown a harmful prompt, two candidate responses, and one of the constitution's principles, it is simply asked which response is less harmful.
From an off-the-shelf model turned into a labeler, the story moves to how its judgments come to supply half of a hybrid reward signal, connecting onward to the comparison format it reads from and the preference model it feeds.
An ordinary language model repurposed as a labeler
The feedback model is, in the paper's own words, "an independent model, typically a pretrained LM" (constitutional-ai, §"4.1 Method", p. 10) — nothing architecturally special. It is shown a harmful prompt, a pair of candidate responses, and a principle sampled from Principles for RL-CAI (Appendix C.2), formatted as a Multiple-Choice Comparison Format question, and the log-probability it assigns to each option becomes a soft preference label (Multiple-Choice Comparison Format — turns into a preference labeler → Feedback Model). When chain-of-thought prompting is added, the paper swaps in a Helpful RLHF Model in place of the plain pretrained model, "which typically writes higher quality chain-of-thought" (constitutional-ai, §"4.1 Method", p. 11) — so "the feedback model" names a role filled by different underlying models depending on whether reasoning is elicited.
Supplying half of a hybrid reward, robustly
As the mechanism at the center of RL from AI Feedback (RLAIF), the feedback model supplies the harmlessness half of the Hybrid Human/AI Preference Model, while human labels continue to supply the helpfulness half (Feedback Model — supplies the harmlessness half of → Hybrid Human/AI Preference Model) — 182,831 AI-generated harmlessness comparisons mixed with 135,296 human helpfulness comparisons (constitutional-ai, §"4.2 Datasets and Training", p. 11). Its judgments are made more robust by Principle Ensembling, sampling a different principle per label rather than reusing one, which the paper reports gives "notably more robust PM behavior" (constitutional-ai, §"4.1 Method", p. 11). The feedback model's usability rests on Model Calibration (well-calibrated multiple-choice probabilities) — the pre-existing finding that language-model probabilities on multiple-choice questions are reasonably trustworthy — which is what justifies treating its raw log-probabilities as training targets at all, and which the paper verifies directly on its own HHH Evaluation Set (binary comparisons) questions (constitutional-ai, §"4.3 Main Results", Figure 9, p. 12).