Principle Ensembling
Constitutional AI: Harmlessness from AI Feedback — introduced
Randomly sampling a different constitutional principle for each comparison label (rather than reusing one principle for all labels) to produce more robust preference-model behavior.
Reusing the same principle for every comparison would make the resulting behavior only as robust as that one rule. Instead, this technique draws a fresh principle from the constitution each time, on the idea that variety builds sturdier behavior than any single fixed instruction could.
One technique, applied twice in the supervised stage and the AI-feedback stage with a different payoff each time, runs through the page, which connects to the SL-CAI principle list and the feedback model it serves.
One technique, sampled twice, two different payoffs
Principle ensembling means drawing a different principle from Constitution (set of principles) for each comparison or revision, instead of reusing one principle throughout. The paper applies it at both of CAI's stages, and the payoff differs each time. In the supervised stage, varying which of the Principles for SL-CAI (Appendix C.1) is used at each revision step has almost no effect on harmlessness preference-model scores — results for N = 1, 2, 4, 8, 16 principles are "essentially flat" (constitutional-ai, §"3.4 Scaling Trends", p. 9) — but it does make revised responses more varied, and that diversity improves exploration once the revisions seed the RL stage (Principle Ensembling — diversifies use of → Principles for SL-CAI (Appendix C.1)). In the RL stage, sampling a different one of the sixteen Principles for RL-CAI (Appendix C.2) for every comparison label the Feedback Model produces "led to notably more robust PM behavior compared to using the same principle for all labels" (constitutional-ai, §"4.1 Method", p. 11; §"4.3 Main Results", p. 13), a direct improvement in the thing being ensembled over rather than a deferred one.
Two different homes in the theme structure
This split payoff is why ensembling-principles sits in both Constitutional AI's method: principles, critique, and revision, as one of several validated design choices in the supervised pipeline, and Engineering a language model into a usable preference labeler, as one of the narrow prompting fixes that make a plain language model usable as a preference labeler. The The same technique, cited or deployed twice, doing different work each time connective theme reads the pair of edges together as a small lesson: identical machinery, applied at two different pipeline stages, can fail its most obvious benchmark in one place while succeeding directly in another.