Hybrid Human/AI Preference Model
Constitutional AI: Harmlessness from AI Feedback — introduced
A preference model trained on a mixture of human-labeled helpfulness comparisons and AI-labeled harmlessness comparisons.
This preference model learns from two different sources of judgment at once: people rating which response is more helpful, and an AI rating which is more harmless. Nothing about its architecture changes to make that mixture work, only where the comparisons come from.
What the model is gets explained first, set within a longer line of work on scaling up oversight, before connecting to the feedback model supplying its AI-generated half.
What it is
The hybrid human/AI preference model is architecturally an ordinary reward model — a scalar-output model trained with a pairwise ranking loss — with nothing changed about its structure or training procedure. Given a prompt $x$ and a preferred/dispreferred response pair $(y_w, y_l)$, it is trained with the standard Bradley-Terry preference loss: $$\mathcal{L}(\theta) = -\mathbb{E}_{(x,y_w,y_l)\sim D}\left[\log \sigma\big(r_\theta(x,y_w) - r_\theta(x,y_l)\big)\right].$$ The innovation is entirely in what $D$ is built from: 135,296 human-labeled helpfulness comparisons stitched together with 182,831 harmlessness comparisons generated entirely by the feedback model, one per SL-CAI prompt (constitutional-ai, §"4.2 Datasets and Training", p. 11; introduced at §"1.2 The Constitutional AI Approach", p. 5). Once trained, the resulting model cannot itself distinguish which half of its training data any given judgment traces back to; helpfulness and harmlessness are fused into a single scalar reward.
Its place in the scalable-oversight lineage
This concept is the specific mechanism by which Constitutional AI keeps helpfulness supervision human while making harmlessness supervision synthetic, and it is best read as the point where two separate supervision lineages converge: the human-labeled comparisons descend from ordinary RLHF, while the AI-labeled half is what the feedback model supplies as its output (Feedback Model — supplies the harmlessness half of → Hybrid Human/AI Preference Model). It is a direct descendant, in spirit, of Reward model (RM) generally, inheriting its architecture and loss wholesale while changing only where the training signal originates. The concept sits inside the From an open problem to reward models to AI feedback theme, which traces Concrete Problems' 2016 naming of scalable oversight as an open problem through InstructGPT's reward model to this hybrid design, and inside the Each pipeline stage's output is the next stage's engineered raw material connective theme, since it is the specific stage where two differently-provenanced comparison datasets are stitched into one training signal for the RL stage that follows.