Engineering a language model into a usable preference labeler
Turning a plain language model into a trustworthy source of preference labels takes real engineering, not just a good prompt. This theme covers that work inside Constitutional AI (2022): a multiple-choice format for eliciting judgments, chain-of-thought reasoning to sharpen them, and clamping to tame the overconfidence that reasoning tends to introduce.
Each piece of that engineering is worked through in turn, with model calibration running underneath all of it, the quiet assumption that a model's raw probabilities are already reasonably well-behaved.
RL-CAI's reinforcement learning stage needs preference labels, and this theme covers the specific prompting and calibration engineering that lets a plain feedback model supply them without human input. The multiple-choice comparison format presents a conversation, a constitutional principle, and two labeled candidate responses, from which the feedback model's answer log-probabilities become the preference label. Chain-of-thought prompting, including the specific let's think step-by-step zero-shot trigger and a set of hand-written chain-of-thought few-shot examples, elicits step-by-step reasoning before the model commits to that judgment, making it more accurate and more transparent. Because chain-of-thought reasoning tends to push those probabilities toward overconfidence, label clamping compares raw soft labels, hard 0/1 labels, and probabilities clamped to a fixed range as three ways of constructing usable training targets, relying on the underlying property of model calibration to make any of this meaningful. Principle ensembling, sampling a different principle per label, further diversifies what the feedback model is asked to judge. Each piece here solves a narrow engineering problem in the same larger project: making an AI's judgment usable as training signal.