Semi-supervised reinforcement learning
Concrete Problems in AI Safety — introduced
A proposed framework for scalable oversight in which an RL agent observes its true reward on only a small fraction of timesteps/episodes but is still evaluated on all of them.
Imagine an agent that is told how well it did only once in a while, yet is judged on everything it does. Semi-supervised reinforcement learning is the framework Concrete Problems in AI Safety (2016) builds around that setup, its main technical route to oversight that cannot be given constantly.
The prose below lays out the fraction-of-episodes setup, then works through the approaches the paper sketches for closing the gap between what the agent sees and what it is judged on.
Reward on a fraction of episodes
Semi-supervised reinforcement learning, proposed by Concrete Problems as its central technical answer to Scalable oversight, is a framework that resembles ordinary RL except that the agent observes its true reward on only a small fraction of timesteps or episodes, while its performance is still evaluated on all of them. A baseline can be defined by simply discarding the unlabeled episodes and running an ordinary RL algorithm on what remains; the paper notes this baseline learns very slowly, and the actual challenge is making productive use of the unlabeled episodes to approach, as closely as possible, the learning speed of a fully labeled setting (concrete-problems, §"5 Scalable Oversight", pp. 11-12). The framing draws heavily on an informal essay by one of the paper's authors, and includes both a random-labeling variant and an active-learning variant in which the agent itself requests which episodes to see reward for.
Four proposed instantiations
The paper offers four concrete techniques as children of this concept, none mutually exclusive. Supervised reward learning trains a model to predict reward from state and uses that prediction, weighted by confidence, to estimate payoff on unlabeled episodes; Semi-supervised or active reward learning adds the ability to request labels for salient events specifically; Unsupervised value iteration and Unsupervised model learning both ignore reward prediction entirely and instead extract information from the observed state transitions of unlabeled episodes, sharpening Bellman updates or a learned dynamics model respectively — the edge Unsupervised value iteration — is mirrored in model based form by → Unsupervised model learning shows these last two are the identical extraction move applied to RL's two dominant paradigms, model-free and model-based. The proposed testbed, per Atari games (as an RL testbed) — is the proposed testbed for → Semi-supervised reinforcement learning, is Atari, chosen specifically because it lets the paper stage a graded ladder of oversight-scarcity conditions — from a small number of direct reward signals down to a handful of explicit reward queries — inside one familiar domain (concrete-problems, §"5 Scalable Oversight", pp. 12-13). The concept is a child of Scalable oversight, anchors the Scaling oversight by predicting reward on unlabeled episodes theme and the Turning an abstract oversight worry into dials and pools that can run together connective theme, and belongs to the Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline supertheme.