Unsupervised model learning
Concrete Problems in AI Safety — introduced
A semi-supervised RL approach that uses observed transitions from unlabeled episodes to improve a model-based RL agent's dynamics model.
A model-based agent needs more than reward; it needs to know how the world responds to what it does, and that part of the picture stays visible even when reward does not. Unsupervised model learning, the fourth of the semi-supervised proposals in Concrete Problems in AI Safety (2016), uses reward-free episodes to sharpen exactly that response model.
What follows covers sharpening the dynamics model itself, then how this proposal mirrors, in model-based terms, its unsupervised-value-iteration sibling.
Sharpening a dynamics model without reward
Unsupervised model learning, proposed by Concrete Problems, is one of the paper's four instantiations of Semi-supervised reinforcement learning: for an agent using model-based RL, it uses the observed state transitions of unlabeled episodes — episodes for which no true reward was ever revealed — to improve the quality of the agent's learned dynamics model (concrete-problems, §"5 Scalable Oversight", p. 12). Because a transition $(s,a,s')$ carries information about how the environment moves independent of what it pays, this technique can extract useful signal even from episodes that contribute nothing to reward estimation directly.
The model-based mirror of unsupervised value iteration
The concept exists as a direct structural twin of Unsupervised value iteration. The edge Unsupervised value iteration — is mirrored in model based form by → Unsupervised model learning describes both as the same underlying idea — extract information from unlabeled transitions despite not knowing their reward — applied to reinforcement learning's two dominant paradigms: the value-iteration version presupposes a model-free agent and sharpens Bellman updates directly, while this concept presupposes a model-based agent and sharpens the dynamics model it plans through. Which of the two a builder reaches for is dictated entirely by whether the underlying RL agent is model-free or model-based, not by anything specific to the oversight setting itself; the two are companions covering the same design space rather than competitors. The concept is a child of Semi-supervised reinforcement learning, sits in the Scaling oversight by predicting reward on unlabeled episodes theme and both the Two things built identically, pointed in opposite directions and Turning an abstract oversight worry into dials and pools that can run together connective themes, and belongs to the Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline supertheme.