Preference Label Clamping (Soft vs. Hard vs. Clamped)
Constitutional AI: Harmlessness from AI Feedback — introduced
A technique for constructing PM training targets from feedback-model outputs, comparing raw soft (normalized log-probability) labels, hard 0/1 labels, and probabilities clamped to a fixed range (e.g. 40-60%) to counteract CoT-induced overconfidence.
Chain-of-thought reasoning makes a model's answers sharper, but it also makes its stated confidence swing to extremes, which is bad news if that confidence is meant to be a training signal. This page covers the fix: squeezing the resulting probabilities back into a narrower, more moderate range before using them.
It compares three ways of turning a judgment into a training label, then settles on the right range to clamp to, before connecting back to chain-of-thought prompting as the source of the problem it solves.
Correcting overconfidence introduced by chain-of-thought
Preference label clamping compares three ways of turning a Feedback Model's output into a preference-model training target: raw soft labels (normalized log-probabilities), hard 0/1 labels, and probabilities clamped to a fixed range. Without chain-of-thought reasoning, soft labels work well on their own, because they are "actually fairly well-calibrated" (constitutional-ai, §"4.3 Main Results", p. 13). But Chain-of-Thought (CoT) Prompting breaks this: a CoT explanation "typically states explicitly which multiple choice option is to be preferred, and so the probability targets are typically very confident (i.e., close to 0 or 1) and are not well-calibrated" (constitutional-ai, §"Chain-of-Thought Prompting", p. 11; Chain-of-Thought (CoT) Prompting — creates the overconfidence that necessitates → Preference Label Clamping (Soft vs. Hard vs. Clamped)). Clamping forces the probability into a fixed band instead of using it raw: writing $p$ for the raw probability, the clamped label is $$ p_{\text{clamped}} = \min(\max(p,\,0.4),\,0.6). $$
Finding the right band
The paper tested clamping at 20-80% and found it "slightly improved results," then tested 40-60% and found it "improved results further," settling on 40-60% for the main paper (constitutional-ai, §"4.3 Main Results", p. 13). The effect is not cosmetic: "without the clamping, RL-CAI models would learn to output more extreme responses" (constitutional-ai, §"Chain-of-Thought Prompting", p. 11). As a child of RL from AI Feedback (RLAIF), label-clamping exists only because chain-of-thought was added to the feedback pipeline for accuracy and legibility, and it belongs to the Engineering the exact prompt or format needed to manufacture a model behavior connective theme as the corrective half of that trade — reasoning improves the judgment but degrades the calibration the labeling scheme needs, and clamping repairs the calibration without discarding the reasoning.