Replacing the human in the loop at exactly the point it was the bottleneck
Ten written principles end up doing the job of tens of thousands of human comparisons, and that trade is not a one-off: this corpus swaps a machine judgment in for a human one, at the exact point human labor became the bottleneck, again and again.
The clearest instance opens the page, a feedback model stepping into RLHF's human-labeler role, and from there the pattern loops backward to Concrete Problems' own trusted policy oversight, written six years before RLAIF existed, through red teaming's own shift from manual practice to an automated pipeline, and forward to 2023, where the same substitution spreads further still, to a model grading another model's writing and to a model authoring the training data in the first place.
Constitutional AI's central engineering move recurs at every stage: an AI process substitutes for a human one at precisely the point human throughput was scarce, while the surrounding machinery stays untouched. RLHF swaps the labeler in RLAIF, an explicitly "surgical" modification that leaves preference-model training and RL exactly the same. RLAIF delegates harmlessness comparisons to the feedback model, and this substitution is also what RLAIF makes newly automatable: iterated online training, a refresh loop that no longer needs continuous human throughput. The constitution replaces human harm labels in Constitutional AI outright, trading tens of thousands of comparison labels for on the order of ten natural-language principles. Concrete Problems anticipated the shape of this move six years earlier: human oversight's scalability limit motivates trusted policy oversight, replacing a human gatekeeper with a policy that can render a recoverability judgment at machine speed. Red teaming is the manual process that scales into automated red teaming, removing the same kind of human-labor ceiling. A year later, CAA extends the same substitution past training into evaluation and dataset authorship, points the earlier instances never touched. GPT-4 rates every open-ended generation against a behavior-specific rubric, the same judge-for-human-rater move RLAIF made for harmlessness comparisons, though the paper still spot-checks GPT-4's scores by hand rather than trusting them outright. GPT-4 also authors CAA's refusal dataset (and, the same way, its hallucination dataset) from scratch, standing in for the human writers used for the paper's other five behaviors, one model manufacturing training material that steers a second model whose weights it never sees. The pattern's reach has widened over three years: an AI process first replaced the human labeler supplying preference comparisons in 2022, and by 2023 the identical substitution has spread to the evaluator scoring the outputs and to the writer authoring the material those outputs are built from. By 2025 the same substitution recurs twice more, at scales smaller and larger than either 2022 instance. Claude generation tooling, in the form of Claude 3.7 Sonnet, authors the artifacts behind trait-artifact generation exactly the way GPT-4 authored CAA's refusal dataset, one model manufacturing material another process will later score or steer against; but this call is the cheapest job the same tool performs in the paper, not a representative one, since the same model is elsewhere tasked with generating roughly 5k questions per trait across 50 domains for the Evil and Hallucination finetuning datasets and three response versions apiece for every question in every dataset in the paper, tens of thousands of individual generations dwarfing the single one-shot call that produces a whole trait's evaluation apparatus. And trait-expression-score operationalizes its rubric through GPT-4.1-mini exactly the way open-ended generation evaluation delegated scoring to GPT-4, an AI judge substituting for the human rater time behavior-level evaluation would otherwise cost at scale; but where GPT-4 sampled a single output token per CAA rating, GPT-4.1-mini's score is a logit-weighted sum over the top-20 candidate tokens restricted to the integers 0-100, a continuous readout the paper validates directly against two human judges across 300 pairwise comparisons, 97 percent agreement for evil, 92 for sycophancy, 95 for hallucination, 94.7 percent overall, a paper-specific agreement figure CAA's own GPT-4 rater never reported. The same GPT-4.1-mini also serves elsewhere in the paper as an LLM-based data filter, a separate substitution for a separate human bottleneck, screening rather than rating.