Chain-of-Thought Prompts for RL-CAI (Appendix E.2)
Constitutional AI: Harmlessness from AI Feedback — introduced
Hand-written few-shot examples, each with a substitutable principle placeholder, used to prompt the feedback model to produce step-by-step reasoning before its multiple-choice harmlessness judgment.
A generic instruction to reason step by step only goes so far; the feedback model is also given a handful of fully worked examples, each showing the reasoning pattern with a slot left open for whichever principle is being applied. This page is about those worked examples.
It explains what the examples contain and how they work alongside the simpler generic trigger phrase, then connects to that trigger.
What it is
Introduced by Constitutional AI in Appendix E.2 (constitutional-ai, §"E.2 Chain-of-Thought Prompts for RL-CAI", p. 31): a set of hand-written few-shot examples used to prompt the feedback model, each consisting of a pre-written conversation, a constitutional principle with a substitutable placeholder, two labeled candidate responses, and a fully worked chain-of-thought explanation ending in a multiple-choice judgment, all formatted identically to the live comparisons the feedback model is actually asked to judge.
Its job alongside the generic trigger
As a child of Chain-of-Thought (CoT) Prompting, its contribution is to supply exactly the domain-specific scaffolding that the generic zero-shot trigger "Let's think step-by-step" zero-shot CoT trigger cannot: what a harmlessness judgment's reasoning should look like, what a constitutional principle is, and how a conversation and two labeled responses should be laid out. The paper uses both in a single prompt, never either alone: "we use the 'Let's think step-by-step' prompt...to elicit the chain-of-thought. In addition, we prepend several hand-written, few-shot examples in the same format" (constitutional-ai, §"Chain-of-Thought Prompting", p. 11) ("Let's think step-by-step" zero-shot CoT trigger — is paired with → Chain-of-Thought Prompts for RL-CAI (Appendix E.2)). The trigger phrase starts the reasoning; these examples shape it into something usable as a preference label. The concept sits inside the Engineering a language model into a usable preference labeler theme, which covers the full chain of prompting and calibration choices that make an otherwise plain language model usable as a harmlessness labeler, and the Engineering the exact prompt or format needed to manufacture a model behavior connective theme.