Semi-supervised or active reward learning — gets its first deep rl implementation in → Uncertainty-Based Query Selection (Ensemble Variance)

Concrete Problems (2016) names 'semi-supervised or active reward learning' as a research direction: combine reward prediction with active learning so an agent can flag salient events and request feedback specifically on them (concrete-problems, §"5 Scalable Oversight", p. 12). The 2017 paper, co-authored by two of the same researchers (Christiano and Amodei), gives this proposal one of its first concrete deep-RL forms: sample a large batch of candidate segment pairs, have every member of a reward-predictor ensemble predict a preference for each, and query the human only about the pairs where the ensemble disagrees most (deep-rl-human-prefs, §"2.2.4 Selecting Queries", p. 6). The 2017 paper is candid that this ensemble-variance heuristic is 'a crude approximation' of the ideal, and its own ablation shows the heuristic sometimes performs worse than picking queries at random. So the gap between the 2016 proposal and its 2017 realization is real: what was pitched as 'identify salient events' became, in practice, 'measure ensemble disagreement,' a much cruder proxy that the authors themselves flag as needing improvement.