Scaling oversight by predicting reward on unlabeled episodes
Checking an agent's true objective on every example can be too expensive, and Concrete Problems (2016) answers with semi-supervised reinforcement learning: let the agent see its real reward only occasionally, learning to predict it the rest of the time. One of its four proposals, active reward learning, sat unimplemented until the 2017 paper gave it concrete form, uncertainty-based query selection, asking a human about whichever pair a reward-predictor ensemble disagrees about most.
The walk covers all four 2016 proposals in turn, then follows the one that got built, closing on Atari, the toy testbed proposed here and used by the 2017 paper a year later.
Scalable oversight is the problem of ensuring safe behavior when the true objective is too costly to evaluate on every training example. This theme covers Concrete Problems' (2016) central technical answer: semi-supervised reinforcement learning, a framework in which an agent observes its true reward on only a small fraction of timesteps but is evaluated throughout, instantiated through four concrete proposals. Supervised reward learning trains a model to predict reward from state and uses that prediction, weighted by its own uncertainty, to estimate payoff on unlabeled episodes. Semi-supervised or active reward learning adds the ability to flag salient events and request feedback on them specifically; the 2017 paper gives this proposal its first concrete deep-RL implementation, choosing which segment pairs to query through uncertainty-based query selection, sampling candidates and asking the human about whichever pair a reward-predictor ensemble disagrees about most. Expected-value-of-information query selection is the more principled alternative that the 2017 paper leaves to future work rather than implementing. Unsupervised value iteration and unsupervised model learning both extract information from the transitions of unlabeled episodes alone, improving Bellman updates or a dynamics model respectively without needing reward labels at all. Atari games served as the toy testbed proposed here, and became, the following year, one of the two domains in which the 2017 paper actually ran these ideas. The shared claim is that a cheap, sparse true-reward signal can be stretched to cover dense training data by learning to predict it rather than evaluating it everywhere, a claim the 2017 paper is the first to test at deep-RL scale.