Jailbreaks
Steering Llama 2 via Contrastive Activation Addition — inherited
Adversarial or unusual inputs users find to make an LLM output content its safety training was meant to prevent. Mentioned as motivation for the proposed CAA-based red-teaming application, since unwanted behaviors that persist despite RLHF/finetuning are hard to find systematically.
RLHF and finetuning are supposed to make a model reliably decline to produce harmful output, but "reliably" keeps turning out to have exceptions. Users keep finding prompts, oddly phrased, layered with fictional framing, whatever works, that get a supposedly safety-trained model to say the thing it was trained not to say.
Jailbreaks are the motivating problem behind CAA's proposed red-teaming application: if finding these adversarial prompts by hand is slow and unsystematic, an activation-based shortcut for surfacing the same failures might do the job faster.
Inputs that get around safety training
Jailbreaks are adversarial or unusual inputs that users discover, typically through ordinary use rather than any systematic search, that make a language model output content its safety training was meant to prevent (contrastive-activation-addition, §"9.1 Suggested future work", p. 9). They range from directly adversarial prompt phrasing to more indirect social-engineering framings, such as asking a model to roleplay an unrestricted persona, embedding a disallowed request inside a fictional scenario, or splitting an instruction into steps no single one of which looks harmful. What unifies them is not technique but outcome: each finds some path through a model's input space that finetuning and RLHF did not anticipate, and so did not suppress.
Why they exist, and why that matters for safety validation
Jailbreaks exist because alignment training, whether RLHF or Constitutional AI's supervised-then-RL pipeline, shapes a model's typical behavior under typical inputs; it cannot verify behavior under every possible one. CAA treats their persistence as evidence for a specific claim: that finetuning and RLHF reduce the likelihood of specific dangerous outputs without eliminating the underlying capacity to produce them, since users "can often find jailbreaks to make LLMs output harmful content" even after that training (contrastive-activation-addition, §"9.1 Suggested future work", p. 9). This makes the absence of observed bad behavior weak evidence of its absence in principle - a model never yet jailbroken may simply not have met the right input, not lack the capacity to misbehave.
From an incidental discovery to a target for red-teaming
Red teaming is the deliberate, resourced version of what a jailbreak finds by accident: a model's own developers searching for the same kind of adversarial input before deployment rather than leaving the search to whoever downstream happens to try. Constitutional AI sources its harmful training prompts from crowdworker red teaming and flags automated red-teaming, language models generating the adversarial prompts themselves, as a related future direction. CAA's own proposed red-teaming application goes further, treating jailbreaks not as a problem to solve by finding better prompts but as the motivation for skipping prompts altogether: if unwanted behavior can be triggered by manipulating a model's internal activations directly, that offers a systematic alternative to hunting for jailbreaks by hand.