Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline
No team can review every model output by hand, so an early paper in this corpus sketches ways an AI system might stand in for the missing reviewer. What began as a rough proposal became, years later, an actual training pipeline, and, later still, an object other papers keep returning to probe from new angles.
From that open problem and its early remedies, the page passes through the reward model and AI feedback that grew out of them, into a full pretraining-to-PPO pipeline and the labeling work behind it, and on to where that pipeline still falls short. It closes by following two later papers that treat the finished pipeline as something to examine rather than extend: one asking what an inference-time lever adds on top of it, the other asking what any finetuning run, not only RLHF's own, does to a model underneath the surface.
Concrete Problems (2016) names scalable oversight, evaluating a true objective too costly to check everywhere, as an open problem, then sketches remedies: scaling oversight by predicting reward on unlabeled episodes, and scaling oversight without a learned reward predictor via distant supervision or hierarchical decomposition. The 2017 paper is where those remedies are first built and run, and three themes trace that build: comparisons as the atomic unit of human feedback supplies the mechanism, a human comparing pairs of trajectory segments rather than scoring them, turned into a scalar reward by Bradley-Terry machinery; keeping the reward model on-distribution supplies the countermeasures, online labels, ensembles, regularization, and normalization, that keep that reward trustworthy once a policy optimizes against it; and the economics of human feedback supplies the budget constraint, minutes of non-expert human attention, whose cost accounting made oversight of deep RL look practical for the first time. From an open problem to reward models to AI feedback traces the lineage onward from that 2017 waypoint: the same reward model InstructGPT (2022) scales up, and that Constitutional AI's (2022) RL from AI Feedback pushes a step further by replacing the human labeler with an AI one. InstructGPT's three-step training pipeline lays out that later build end to end, pretraining, supervised fine-tuning, reward modeling, PPO, while the mechanics that make PPO fine-tuning work supplies the engineering beneath its RL stage and building the human side of the RLHF pipeline supplies the labelers and datasets feeding all three steps, a lineage now traced back to the 2017 paper's own preference elicitation protocol and contractors. Learning from demonstrations instead of a specified reward supplies the direct genealogy for the pipeline's own first step, SFT, a technique the 2017 paper itself weighs against preferences and sets aside. Where RLHF alignment still falls short documents the residue left once this pipeline is actually built, hallucination, hedging, and reward over-optimization among them, a residue Steering Llama 2 via Contrastive Activation Addition (2023) shows persists, as sycophancy, in an entirely different RLHF-trained model. That 2023 paper supplies two more lenses on the finished RLHF stack from outside rather than tracing its build. Three levers on one model: prompting, finetuning, steering asks what an inference-time steering vector adds on top of that finished stack beyond system prompting or supervised finetuning, at a fraction of the compute and with better out-of-distribution generalization on at least one behavior finetuning fails to transfer. Base versus chat: what RLHF changes inside turns the same vectors into a probe of the pipeline's product: vectors extracted from Llama 2's base model transfer onto its RLHF-trained chat counterpart, most strongly between layers 7 and 15, evidence that RLHF re-weights a behavior's expression rather than rewriting its representation from scratch. Finetuning moves the persona, measurably picks the pipeline's SFT step back up and asks what any finetuning run, not only RLHF's own, does to a model's trait directions: the 2025 paper shows a training run displaces the model along a trait's direction by an amount, the activation shift, that predicts how strongly the trait shows up once finetuning finishes, and that these shifts move together across traits, so that data narrowly built to elicit one bad behavior tends to drag others along with it, the paper's mechanistic handle on emergent misalignment, the phenomenon in which narrow bad data produces a broadly misaligned model. Where base-versus-chat treats RLHF as a finished product and reads only what it left unchanged, this theme opens a finetuning run up while it happens and measures the displacement directly. The pipeline's arc now runs from a 2016 open problem through a 2023 measurement of what RLHF left unchanged inside the model to a 2025 account of how any finetuning run moves it there in the first place.