Context Distillation
Constitutional AI: Harmlessness from AI Feedback — inherited
A prior finetuning technique of distilling a lengthy prompt's behavior into model weights, used for RLHF models in earlier work but not for the RLHF models trained directly in this paper.
A long, carefully engineered prompt can shape a model's behavior, but keeping that prompt around forever is wasteful. Context distillation instead bakes the prompt's effect directly into the model's weights by finetuning it on its own outputs under that prompt. Earlier RLHF work used the trick; the RLHF models trained directly in this paper deliberately do not.
The technique itself is explained first, then why the paper deliberately sets it aside, closing the discussion within the lens on prior systems Constitutional AI positions itself against.
What it is
Context distillation is a fine-tuning technique that predates Constitutional AI (Askell et al., 2021; Snell et al., 2022). A model is fine-tuned on its own completions, generated under a long, carefully engineered prompt or instruction context, so that the behavior the prompt induces is absorbed into the weights themselves via ordinary supervised fine-tuning; the prompt can then be discarded at inference time, since the model has already learned to act as if it were still present.
A deliberate non-use
Constitutional AI is explicit that this concept is a comparison point by contrast rather than a component of its own pipeline. Bai et al.'s (2022) earlier RLHF models were fine-tuned starting from context-distilled checkpoints, but Constitutional AI's own RLHF models are fine-tuned directly from pre-trained models instead: "we didn't see much benefit to using context distillation since the improvement from RL was much more significant" (constitutional-ai, §"4.2 Datasets and Training", p. 11). It sits in the Prior dialogue-agent systems Constitutional AI positions itself against theme alongside LaMDA and Sparrow as one of the paper's explicit points of comparison against prior aligned-dialogue-agent work, though unlike those two it names a technique rather than a system — the paper's point in mentioning it is methodological, marking what it chose not to do and why.