Unsupervised value iteration
Concrete Problems in AI Safety — introduced
A semi-supervised RL approach that uses observed transitions from unlabeled episodes to make more accurate Bellman updates.
Even an episode with no reward attached still shows how the world moves. Unsupervised value iteration, one of four proposals in Concrete Problems in AI Safety (2016), puts those unlabeled transitions to work, using them to sharpen the value estimates an agent keeps through Bellman updates.
The first section shows how those updates get made; the second sets the idea beside its companion and its complement among the sibling proposals.
Bellman updates from unlabeled transitions
Unsupervised value iteration, proposed by Concrete Problems, is one of the paper's four instantiations of Semi-supervised reinforcement learning. It uses the observed transitions of episodes for which no true reward was ever revealed to make more accurate Bellman updates (concrete-problems, §"5 Scalable Oversight", p. 12). In an ordinary model-free setting, a value function is refined by an update such as $$V(s) \leftarrow V(s) + \alpha \big[r + \gamma V(s') - V(s)\big],$$ which ordinarily requires the observed reward $r$; the unsupervised version of this idea extracts what it can from the transition $(s, s')$ itself — information about how the environment moves — even on episodes where $r$ is never observed, leaving reward-dependent refinement to whatever labeled data the agent does have access to.
Its companion and its complement
The concept has two edges, each defining a different kind of relationship. Against Supervised reward learning, the edge Unsupervised value iteration — extracts a complementary signal to → Supervised reward learning shows the two techniques mining the identical pool of unlabeled episodes for non-overlapping information: reward-shaped information from states, versus dynamics-shaped information from transitions, meaning nothing in the paper's framing prevents running both on the same batch of data simultaneously. Against Unsupervised model learning, the edge Unsupervised value iteration — is mirrored in model based form by → Unsupervised model learning treats the two as the same idea applied to RL's two dominant paradigms, model-free versus model-based, with the choice between them dictated entirely by which kind of agent architecture a builder is already using. The concept is a child of Semi-supervised reinforcement learning, sits in the Scaling oversight by predicting reward on unlabeled episodes theme and both the Two things built identically, pointed in opposite directions and Turning an abstract oversight worry into dials and pools that can run together connective themes, and belongs to the Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline supertheme.