Engineering the exact prompt or format needed to manufacture a model behavior
Ask a model to explain itself before answering, and the explanation can quietly break the very calibration an earlier paper's labeling scheme depends on. The feedback model that makes that scheme possible is held together by prompting choices this narrow and this fragile, and later papers turn out to depend on comparably narrow choices of their own.
The page starts with calibration and works outward: why trustworthy probabilities let a plain multiple-choice format double as a preference labeler, how asking for reasoning first undoes that trust, and how a later steering paper borrows the same A-or-B format for an unrelated job, reading off internal activations instead of a probability. It closes on the corpus's newest paper, where a monitoring tool depends on its own calibration choices, which layer to read, and how its correlation with behavior holds up once the many prompts and doses used to elicit that behavior are examined separately rather than pooled together.
Constitutional AI's feedback-model pipeline depends on a chain of narrow, load-bearing prompting choices. Model calibration is the precondition for the multiple-choice evaluation format, since converting raw log-probabilities into soft preference labels is only meaningful if those probabilities are trustworthy; the same multiple-choice evaluation format is what turns an ordinary pretrained LM into a preference-labeling feedback model. Model calibration is later empirically verified on the HHH eval benchmark, closing the loop between assumption and evidence. But chain-of-thought prompting creates the overconfidence that necessitates label clamping, since a CoT explanation stating its answer up front breaks the calibration the format depends on, forcing probabilities to be clamped into a 40-60% band. The let's-think-step-by-step prompt is paired with hand-written RL-CAI CoT prompts because triggering reasoning and shaping its content are different jobs. And SL-CAI's few-shot examples exist purely to prevent role confusion in critique-and-revision, keeping one model from blurring its critic and reviser roles within a single context. A year later, CAA's multiple-choice contrast-pair format repurposes this same AB surface mechanic for an unrelated job: instead of reading a feedback model's log-probability over 'A' or 'B' out as a preference label, it extracts residual-stream activations at the token position of the answer letter and differences them across matched pairs to build a steering vector. The format cancels confounders for that construction precisely because a pair's two prompts are identical up to the final letter, so phrasing and topic contribute equally to both activations and cancel in the average, leaving only the behavior-relevant component; an appendix corroborates that the letter is doing real work by showing a model conditioned on having 'chosen' A or B spontaneously justifies that choice in its continuation, evidence the single token switches the model into a genuinely different behavioral mode rather than a superficial one. Persona vectors' own monitoring apparatus depends on an analogous chain of calibration choices. Persona-vector computation sweeps and selects a layer within the residual stream rather than committing to one in advance, averaging response-token activations at every layer and only afterward testing steering effectiveness across the full stack to pick the most informative layer; Appendix B.4 shows why the sweep is necessary rather than a formality, trait expression score as a function of steering coefficient varying substantially across layers 5 through 25 for both models tested, no single layer dominating uniformly across evil, sycophancy, and hallucination, and the two base models not even swept with the same coefficient range, Qwen's sweep running to 2.5 and Llama's only to 1.2, since how far a given layer tolerates perturbation before losing coherence is itself model-specific. Once a layer and vector are fixed, projection-based persona monitoring is calibrated against system-prompting as one of two elicitation axes, and the calibration exposes an uneven picture: Table 2's overall correlations, r=0.747 for evil, 0.798 for sycophancy, 0.830 for hallucination, look strong, but the within-condition correlations, computed separately within each of the eight interpolated system prompts and then averaged, drop sharply for two of the three traits, to 0.511 for evil and just 0.245 for hallucination, revealing that hallucination's headline correlation is mostly a between-condition artifact, the projection distinguishing which of the eight prompts was used far better than it distinguishes graded severity within a fixed prompt. Many-shot prompting supplies the second elicitation axis for the same monitor, and it is a structurally different instrument than the first: where system-prompting elicits categorically across eight discrete prompts, many-shot elicits by dose, 0, 5, 10, 15, or 20 exemplars of the target trait, and that graded structure changes what its correlation measures, its within-condition correlation staying close to its overall correlation for evil, 0.735 versus 0.755, and sycophancy, 0.813 versus 0.817, unlike system-prompting's much larger gaps on the same two traits, though hallucination remains the exception, many-shot's overall correlation there, 0.634, the lowest of any trait-method pairing in the table. Layer selection, system-prompt calibration, and many-shot calibration are three separate load-bearing choices behind one correlation figure, each with its own failure mode the paper's own appendix tables expose.