Turning an abstract oversight worry into dials and pools that can run together
An oversight budget sounds abstract until Concrete Problems turns it into a literal dial: the fraction of episodes that get a human label, tunable and testable like any other hyperparameter.
Semi-supervised reinforcement learning is where that dial lives, with Atari and bAbI proposed as domains graded specifically to stress-test it at different settings. The page ends on a neat efficiency: two separate learning signals pulled from the exact same batch of unlabeled episodes, at no extra cost.
Concrete Problems converts the philosophical worry behind scalable oversight into concrete, combinable experimental machinery rather than a single technique. Scalable oversight is operationalized as semi-supervised reinforcement learning, turning an oversight budget into a literal labeled-fraction dial; the Atari games environment is proposed as its testbed specifically because it stages a graded difficulty ladder of oversight scarcity in one familiar domain, and bAbI tasks supply the benchmarking template the paper wants an analogous safe-exploration suite to follow. Within that framework, supervised reward learning is extended by semi-supervised active reward learning, adding the ability to request labels for salient events, while unsupervised value iteration extracts a complementary signal to supervised reward learning from the identical pool of unlabeled episodes, dynamics-shaped information rather than reward-shaped, so nothing stops a single agent from running both simultaneously on one batch of data. The 2017 paper is where several of these proposals first meet an actual deep-RL implementation, sometimes faithfully and sometimes only partly. The asynchronous reward-learning architecture gives concrete shape to the RLHF paradigm more broadly, turning "learn a reward model from preferences and optimize it with RL" into a specific answer about scheduling, three loops running concurrently rather than in fixed phases. Semi-supervised active reward learning gets its first deep-RL implementation in uncertainty-based query selection, but only a crude one: the 2016 proposal to have an agent flag salient events for labeling becomes, in practice, three bootstrap-trained predictors voting by variance, a proxy the 2017 authors themselves call crude and which their own ablations show can underperform querying at random. Two of the framework's other candidate mechanisms are instead explicitly set aside rather than implemented: use demonstrations is passed over in favor of the preference elicitation protocol because demonstration requires a human able to perform the behavior, a requirement preferences never impose; and cooperative inverse reinforcement learning is narrowed to a preferences-only channel in that same protocol, trading CIRL's richer repertoire of physically embedded human moves for a channel simple enough to run at scale from a rendered video clip.