Semi-supervised or active reward learning
Concrete Problems in AI Safety — introduced; Deep Reinforcement Learning from Human Preferences — refined
A semi-supervised RL approach combining reward prediction with active or semi-supervised learning so the agent can identify salient events and request reward feedback on them specifically.
Rather than passively accepting whichever episodes happen to be labeled, an agent could ask: it could notice salient events and request reward feedback on those in particular. Concrete Problems in AI Safety (2016) calls this semi-supervised or active reward learning, a refinement of its basic reward-prediction proposal.
The page explains how that choice gets layered on top of ordinary reward prediction, then places the idea among its sibling approaches within the same framework.
Adding active learning to reward prediction
Semi-supervised or active reward learning, introduced by Concrete Problems, extends the paper's baseline proposal for predicting reward on unlabeled episodes by letting the agent itself influence which episodes get the scarce true-reward label. Rather than accepting whatever labels a passive random or fixed schedule happens to supply, the agent learns to identify "salient" events in the environment and requests to see the reward associated specifically with those events (concrete-problems, §"5 Scalable Oversight", p. 12) — combining reward prediction with traditional semi-supervised or active learning to more quickly learn the reward estimator.
Its place in the family
The concept is explicitly framed as an extension rather than an independent proposal. The edge Supervised reward learning — is extended by → Semi-supervised or active reward learning states the paper introduces it with "combine the above with traditional semi-supervised or active learning" (concrete-problems, §"5 Scalable Oversight", p. 12), building directly on Supervised reward learning's architecture: the underlying reward predictor and its use for estimating payoff on unlabeled episodes stay the same, and only the mechanism for allocating scarce labels changes, from passive to agent-directed. The concept is a child of Semi-supervised reinforcement learning, sits in the Scaling oversight by predicting reward on unlabeled episodes theme and the Turning an abstract oversight worry into dials and pools that can run together connective theme, and belongs to the Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline supertheme as the most sample-efficient of the paper's four proposed instantiations of scalable oversight; the following section traces how it became one of the first the very next year's deep-RL paper actually built.
Its first deep-RL implementation: ensemble-disagreement query selection
The 2016 proposal gets one of its first concrete deep-RL implementations the very next year. The 2017 paper selects which trajectory-segment pairs to send a human for labeling "based on an approximation to the uncertainty in the reward function estimator, similar to Daniel et al. (2014)": it samples a large number of candidate pairs, has every predictor in its ensemble predict which segment will be preferred, and queries the pairs where those predictions disagree most across ensemble members (Semi-supervised or active reward learning — gets its first deep rl implementation in → Uncertainty-Based Query Selection (Ensemble Variance); deep-rl-human-prefs, §"2.2.4 Selecting Queries", p. 6). This mechanism is Uncertainty-Based Query Selection (Ensemble Variance), and the paper is explicit that it is a crude approximation to what active-learning theory actually calls for: its own ablation studies show it can even impair performance on some tasks, and the paper names the alternative it does not build, querying by the expected value of information of each candidate comparison (Akrour et al., 2012; Krueger et al., 2016), leaving that direction, Expected-Value-of-Information Query Selection, to future work (deep-rl-human-prefs, §"2.2.4 Selecting Queries", p. 6). The 2017 paper's authorship overlaps with Concrete Problems, Christiano and Amodei appear on both, which is part of why the implementation reuses this concept's canonical id rather than coining a new one.