Red Teaming
Constitutional AI: Harmlessness from AI Feedback — inherited; Steering Llama 2 via Contrastive Activation Addition — inherited
The practice of having crowdworkers conduct adversarial conversations designed to bait a language model into harmful outputs, used as the source of harmful prompts for both SL-CAI and RL-CAI training.
Deliberately trying to break a system before someone else does is a practice with roots well outside AI, in military wargaming and cybersecurity. Here, crowdworkers play the adversary, holding conversations designed to coax a language model into saying something harmful, and those conversations become training material for both stages of Constitutional AI.
The page defines the practice and describes how Constitutional AI runs it, then turns to an automated variant and an outside check, opening onto automated red teaming and connecting to the PALMS dataset, before closing on a 2023 proposal to run the same search inside a model's activations instead of through conversation at all.
What red teaming is
Red teaming is a general adversarial-testing practice that predates Constitutional AI: deliberately adopting an adversarial role against a system to find its failure modes before deployment, with roots in military wargaming and long-standing use in cybersecurity. It was adapted for language models specifically in the companion paper Ganguli et al. (2022), "Red Teaming Language Models to Reduce Harms," which Constitutional AI's own harmful-data pipeline draws on directly.
How Constitutional AI runs it
In this paper, red teaming means paid crowdworkers holding back-and-forth adversarial conversations designed to bait the assistant into generating harmful content, one prompt per human turn (constitutional-ai, §"3.1 Method", p. 7). The resulting prompts source the harmful training data for both stages of the pipeline: 182,831 red-team prompts in total (42,496 human-written plus 140,335 generated by few-shot prompting a pretrained model) build SL-CAI's critique-and-revision data, and a further 491,142 model-generated red-team prompts feed RL-CAI (constitutional-ai, §"3.2 Datasets and Training", p. 8). The same underlying transcripts, together with the crowdworker harm ratings collected alongside them, are reused directly to build Harmful Behavior Identification/Classification Evaluations (Appendix B)'s two multiple-choice tests (Red Teaming — supplies the transcripts for → Harmful Behavior Identification/Classification Evaluations (Appendix B)).
Its child, and its one external check
Manual red teaming has an inherent human-labor ceiling, and its named child, Automated Red Teaming, is the paper's flagged solution: replacing the crowdworker with a language model that generates the adversarial prompts itself, a move the paper explicitly ties to Evasiveness being eliminated first, since an evasive target model would just teach an automated red-teamer to harvest refusals rather than real vulnerabilities (Red Teaming — is the manual process that scales into → Automated Red Teaming). Because every red-team-derived prompt shares one pipeline, PALMS (Process for Adapting Language Models to Society) is used as an independent, externally-authored prompt set precisely because it falls outside this loop (PALMS (Process for Adapting Language Models to Society) — provides an independent check outside → Red Teaming). Red teaming's central role in the Sourcing and classifying harmful prompts theme, and its recurrence across the A primary metric, and a second instrument built to catch what it can't, Replacing the human in the loop at exactly the point it was the bottleneck, and Each pipeline stage's output is the next stage's engineered raw material connective themes, reflects that nearly every harmful-content mechanism in the paper — training data, evaluation, and comparison baseline alike — traces back to this one human-driven process.
2023: an internal alternative to searching input space
Steering Llama 2 via Contrastive Activation Addition (2023) adds a coda to this practice from a different angle than either the manual or automated variants above: rather than search for adversarial inputs at all, it proposes searching the model's activations directly. The paper frames the motivation as a limitation shared by every red-teaming approach in this corpus: "validating if finetuning and RLHF have made models robustly safe is challenging... unwanted behaviors can often still be elicited with adversarial or unusual inputs," and users "can often find Jailbreaks to make LLMs output harmful content," yet "systematically finding inputs that reveal these flaws is challenging" (contrastive-activation-addition, §"9.1 Suggested future work", p. 9). Its proposed alternative keeps the same goal, eliciting unsafe behavior before deployment, but changes the surface being searched: use a steering vector "as an adversarial intervention to trigger unwanted behaviors in models more efficiently," on the premise that "if a behavior can be easily triggered through techniques such as CAA, it may also occur in deployment," and, conversely, that failing to elicit a behavior through a small internal perturbation could serve as a stronger safety guarantee than failing to find it by manual search (contrastive-activation-addition, §"9.1 Suggested future work", p. 9). It is offered only as suggested future work, not a result the paper runs — an internal complement to crowdworker conversations and automated red-teaming's model-generated prompts, not a replacement for either.