Scaling oversight by predicting reward on unlabeled episodes — is selectively realized in → From an open problem to reward models to AI feedback

Of the four semi-supervised instantiations this theme proposes, only one survives, in altered form, into the lineage's reward model. Supervised reward learning trains a model to predict scalar reward from state, weighted by its own uncertainty, so a sparse true-reward signal can be interpolated onto unlabeled episodes; InstructGPT's reward model instead predicts relative preference between two full completions via a pairwise comparison loss (instructgpt, §3.5 Models, p. 8), applied uniformly rather than interpolated onto a labeled/unlabeled split, since RLHF's training pool has no directly labeled episodes at all. Active reward learning's request-feedback-on-salient-events mechanism, unsupervised value iteration, and unsupervised model learning, the theme's other three branches, have no counterpart anywhere in the lineage. The 2016 theme is best read not as a blueprint the reward model followed but as a broader menu from which one general idea, a trained stand-in for costly evaluation, was kept and three specific mechanisms were dropped.