Distant supervision
Concrete Problems in AI Safety — inherited
A prior weak/semi-supervised learning approach providing aggregate statistics or noisy labeling rules instead of per-example evaluations, drawn from NLP and weakly-supervised learning literature.
Long before AI safety borrowed the idea, natural language processing researchers were training systems without labeling each example, supplying aggregate statistics or rules that stamp out many noisy labels instead. Distant supervision names that family of techniques, which Concrete Problems in AI Safety (2016) picks up as one route to cheaper oversight.
What follows traces that trade of per-example evaluation for weaker signal, then follows the branches the paper draws off scalable oversight, including the knowledge-base system it points to as evidence the approach works.
Trading per-example evaluation for aggregate or noisy signal
Distant supervision names a family of techniques from semi-supervised and weakly-supervised learning, developed mainly in natural language processing, in which a system is trained without per-example labels by instead supplying weaker, cheaper information: aggregate population-level statistics, or rules that generate many noisy labels automatically. The approach predates Concrete Problems by several years — it draws on work such as generalized expectation criteria, which asks a user to state statistics like "on average each sentence contains at least one noun" rather than labeling individual sentences, and pattern-extrapolation methods that generalize from an initial set of low-recall labeling rules (concrete-problems, §"5 Scalable Oversight", p. 13). DeepDive, a working rule-based knowledge-base system, is cited as the paper's third concrete example of the same general strategy.
Two branches off scalable oversight
Concrete Problems proposes adapting distant supervision to reinforcement learning as one route to Scalable oversight, complementary to Semi-supervised reinforcement learning but explicitly less experimentally ready: the edge Scalable oversight — is addressed by adapting → Distant supervision notes the paper only goes as far as saying that "expanding these lines of work ... could provide an approach to scalable oversight" (concrete-problems, §"5 Scalable Oversight", p. 13), without the concrete cartpole and Atari experiments proposed for semi-supervised RL. The obstacle is structural: distant supervision's originating literature assumes examples are drawn i.i.d., an assumption an interactive RL agent, which shapes the very data stream it learns from, routinely violates. Set against Hierarchical reinforcement learning (HRL) in the edge Distant supervision — offers a structurally different route than → Hierarchical reinforcement learning (HRL), the two proposals weaken oversight along different axes: distant supervision keeps the fine-grained, per-decision timescale but accepts a weaker signal at each point, while hierarchical RL keeps a strong signal but only requires it rarely, at the top of a delegation hierarchy — cheap-but-weak against infrequent-but-strong. The concept anchors the Scaling oversight without a learned reward predictor theme, sits in the Two or more remedies for the identical problem, sorted by where they intervene connective theme, and belongs to the Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline supertheme as one of two structurally distinct alternatives to reward prediction.