Chain-of-Thought (CoT) Prompting — creates the overconfidence that necessitates → Preference Label Clamping (Soft vs. Hard vs. Clamped)

chain-of-thought-prompting is asked of the feedback model specifically because it improves accuracy and legibility, but it has a side effect the paper had to design around: a CoT explanation almost always states its preferred answer explicitly before the model outputs a probability, so the resulting log-probabilities land close to 0 or 1 instead of remaining softly calibrated. "One issue that arises is that the CoT samples typically state explicitly which multiple choice option is to be preferred, and so the probability targets are typically very confident... and are not well-calibrated" (constitutional-ai, §"Chain-of-Thought Prompting", p. 11). label-clamping exists to correct exactly this: forcing the probability into a fixed band (ultimately 40-60%) so that "without the clamping, RL-CAI models would learn to output more extreme responses" (§"4.3 Main Results", p. 13).