Supervised reward learning
Concrete Problems in AI Safety — introduced
A semi-supervised RL approach that trains a model to predict reward from state, used to estimate payoff on unlabeled episodes with an uncertainty-weighted confidence.
The simplest way to cope with scarce reward is to guess the missing values: train a model to predict reward from what the agent sees, and trust its guesses where no true label exists. Supervised reward learning is the most direct of the four approaches Concrete Problems in AI Safety (2016) offers for its semi-supervised framework.
What follows walks through the reward-predicting model itself, then places it as the base case the other approaches build on or complicate.
Predicting reward from state
Supervised reward learning, proposed by Concrete Problems, is the most direct of the paper's four instantiations of Semi-supervised reinforcement learning: train a model to predict reward from state, either per-timestep or per-episode, and use that prediction, weighted by an appropriate confidence or uncertainty estimate, to estimate the payoff of episodes that carry no true reward label at all. The paper notes this may be more feasible than it first appears, since many existing RL approaches — particularly policy-gradient methods with a strong baseline — already fit value estimators that closely resemble reward predictors (concrete-problems, §"5 Scalable Oversight", p. 12).
Base of the family, complementary to dynamics-based signal
This concept functions as the baseline instantiation the paper's other proposals build on or run alongside. The edge Supervised reward learning — is extended by → Semi-supervised or active reward learning describes Semi-supervised or active reward learning as an extension that changes only how labels are allocated — adding agent-requested labeling for salient events — while leaving the underlying prediction-and-weighting architecture untouched. A separate edge, Unsupervised value iteration — extracts a complementary signal to → Supervised reward learning, shows Unsupervised value iteration mining the same pool of unlabeled episodes for an entirely different, non-overlapping kind of information: where supervised reward learning treats the missing reward as the thing to be estimated, unsupervised value iteration ignores reward altogether and squeezes information from observed state transitions instead, so nothing prevents a single agent from running both techniques simultaneously on the same batch of data. The concept is a child of Semi-supervised reinforcement learning, sits in the Scaling oversight by predicting reward on unlabeled episodes theme and the Turning an abstract oversight worry into dials and pools that can run together connective theme, and belongs to the Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline supertheme.