Scalable oversight
Concrete Problems in AI Safety — introduced; Constitutional AI: Harmlessness from AI Feedback — refined; Deep Reinforcement Learning from Human Preferences — refined
The problem of ensuring safe agent behavior when the true objective function is too costly to evaluate on every training example, forcing reliance on cheaper proxies.
Some objectives are too expensive to check on every training example; no human can grade everything an agent does. Scalable oversight is the problem of keeping behavior safe anyway, working from cheaper stand-ins for the true objective. Concrete Problems posed it in 2016; Constitutional AI (2022) later treated it as a whole family of techniques.
The bare problem gives way to the paper's own proposals, and then to what the idea eventually became: reward models trained to stand in for human judgment, and AI feedback replacing much of the grading itself.
The problem: a true objective too costly to check everywhere
Scalable oversight, introduced by Concrete Problems in 2016, names the difficulty of ensuring safe agent behavior when the objective a designer actually cares about is too expensive to evaluate on every training example, forcing reliance on cheaper proxies instead. The paper's own illustration is its recurring cleaning robot: the objective a designer really wants is something like "if the user spent a few hours looking at the result in detail, how happy would they be?", but training requires relying on cheap approximations like "does the user seem happy when they see the office?" or "is there visible dirt on the floor?" (concrete-problems, §"5 Scalable Oversight", p. 11). Called "scalable supervision" in the paper's abstract, the concept is presented as more than a standalone problem: the divergence between cheap proxy and expensive true objective is said to exacerbate both Negative side effects (avoiding) (which the complex objective might penalize but the cheap approximation omits) and reward hacking (which thorough oversight would recognize as undesirable, but a proxy will not).
The paper's own proposals
Concrete Problems sketches three structurally different routes to scalable oversight, all children of this concept. Semi-supervised reinforcement learning is the paper's primary technical answer, per the edge Scalable oversight — is operationalized as → Semi-supervised reinforcement learning, converting the abstract worry into a literal, tunable labeled-fraction dial that can be dialed from fully supervised toward almost unsupervised and tested on toy domains like cartpole and Atari. Distant supervision and Hierarchical reinforcement learning (HRL) are offered as complementary alternatives that do not depend on training a reward predictor at all — the former making each oversight check cheaper, the latter making checks rarer by restricting the expensive true-objective evaluation to a small number of abstract, top-level decisions. All three live inside the Scaling oversight by predicting reward on unlabeled episodes and Scaling oversight without a learned reward predictor themes.
The first working demonstration: 2017
Concrete Problems sketches routes to scalable oversight but builds none of them at deployed scale; the very next paper in this corpus is where the problem first gets an actual working answer. Its agent is trained to "solve a range of benchmark tasks for deep RL" without observing the true reward, learning the goal of each task only by being asked which of two trajectory segments a human prefers (deep-rl-human-prefs, §"3.1 Reinforcement Learning Tasks with Unobserved Rewards", p. 6). It reports doing so while providing feedback on less than 1% of the agent's interactions with the environment, reducing the cost of human oversight "far enough that it can be practically applied to state-of-the-art RL systems" (deep-rl-human-prefs, §"Abstract", p. 1). The paper backs this with an explicit cost accounting rather than only a sample-efficiency number: the compute for a full Atari training run costs roughly $25, and the roughly 5,000 labels behind it correspond to about 5 hours of contractor time, on the order of $36 at US minimum wage (deep-rl-human-prefs, §"4 Discussion and Conclusions", p. 11). From this the paper concludes that learning a separate reward model can reduce the interaction complexity needed "by roughly 3 orders of magnitude" (deep-rl-human-prefs, §"4 Discussion and Conclusions", p. 10). This is the direct empirical demonstration behind Reward model (RM) and the wider Reinforcement learning from human feedback (RLHF) paradigm this concept's lineage runs through, five years before Constitutional AI gives the same problem its further refinement under the label "Scaling Supervision."
What it became: reward models and RLAIF
This concept has two throughlines running out of 2016: one through 2017, made explicit in the paper's own related work and traced in the previous section, and one through 2022, visible only in hindsight. Neither the 2016 paper nor InstructGPT ever states this second connection directly; it becomes visible only once Constitutional AI revives the term explicitly under the label "Scaling Supervision" and observes that "work on reinforcement learning from human feedback... has already taken a step in the direction of scaled supervision, since the reward signal in RL actually comes from an AI preference model (PM) rather than from immediate human oversight" (constitutional-ai, §"Scaling Supervision", p. 2). Read backward through that later framing, the edge Scalable oversight — is retroactively operationalized by → Reward model (RM) identifies InstructGPT's Reward model (RM) as exactly the cheap, densely-queryable proxy for an expensive true objective that scalable oversight called for six years earlier — a retrospective match documented in the Connections the source papers never draw, visible only by reading them together connective theme and the From an open problem to reward models to AI feedback theme, which traces the fuller arc from reward models through Reinforcement learning from human feedback (RLHF) to Constitutional AI's RL from AI Feedback (RLAIF) and Feedback Model. Within the 2016 paper itself, the concept also sits at the root of the Turning an abstract oversight worry into dials and pools that can run together connective theme, and it gives its name to the entire Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline supertheme — the single clearest case in this corpus of a problem named once, in the abstract, and then genuinely built, first the following year and again, at far greater scale, six years later.