Sourcing and classifying harmful prompts
A model cannot learn to refuse a kind of harm nobody ever tried on it. This theme covers where the harmful training data behind Constitutional AI (2022) actually comes from: crowdworkers deliberately baiting the assistant into bad behavior, a process called red teaming, plus the evaluations built to check whether the resulting harm judgments hold up.
That data's path runs from adversarial conversation to finished training set, on to automated red teaming, the direction the paper points toward next once reduced evasiveness makes it easier for language models to generate their own adversarial prompts, and finally to a 2023 paper's proposal to skip prompts altogether and search for the same unwanted behavior directly inside a model's activations.
Constitutional AI's harmful training data comes from a specific, human-driven process, and this theme covers where it originates and how it is checked. Red teaming has crowdworkers conduct adversarial conversations designed to bait the assistant into harmful outputs, and this is the source of harmful prompts used in training both SL-CAI and RL-CAI. Automated red teaming is flagged as a related future direction, in which language models generate the red-team prompts themselves, and the paper notes that CAI's reduced evasiveness should make scaling this approach easier. Two multiple-choice evaluations, grouped here as harmful-behavior identification and classification, test whether a model can replicate crowdworker harm judgments, distinguishing harmful from ethical assistant behavior and sorting harmful conversations into one of nine categories, built directly from red-teaming transcripts. PALMS, a prior values-targeted dataset of sensitive questions, supplies an independent fixed prompt set used to qualitatively compare HH RLHF and RL-CAI responses. Jailbreaks, the adversarial or unusual inputs ordinary users find on their own to make a model output content its safety training was meant to prevent, are what red teaming is trying to source systematically and in advance: Steering Llama 2 via Contrastive Activation Addition (2023) cites them as the reason ad hoc, deployment-time discovery of unwanted behavior is unreliable, and proposes its own steering vectors as a more systematic alternative for eliciting the same kind of failure under controlled conditions before deployment. Together these concepts describe the harm-data supply chain underneath Constitutional AI's harmlessness claims, and the adversarial-discovery problem that supply chain is built to get ahead of.