Scaling oversight without a learned reward predictor
Predicting a reward is not the only way to make scarce oversight go further. This theme covers two other routes Concrete Problems (2016) sketches: borrowing distant supervision from NLP, where noisy aggregate rules substitute for per-example judgment, and hierarchical reinforcement learning, which delegates oversight down a chain of agents working at different timescales.
The two routes sit side by side here, meeting at feudal reinforcement learning, the concrete architecture behind the hierarchical idea, where managers set goals that subordinates pursue with their own, denser rewards.
Alongside semi-supervised reward learning, Concrete Problems sketches two other routes to scalable oversight that do not depend on training a model to predict reward. Distant supervision, drawn from NLP and weak-supervision literature, substitutes aggregate statistics or noisy labeling rules for per-example evaluation; DeepDive is cited as a working system built on exactly that principle, asking users to supply rules that generate many weak labels rather than hand-labeling data directly. Hierarchical reinforcement learning takes a different route entirely, decomposing oversight across levels of abstraction: a top-level agent issues abstract, long-timescale goals that sub-agents pursue using synthetic, denser reward signals down to primitive actions, tracing back to the feudal reinforcement learning architecture in which managerial agents set goals for subordinates. This theme's claim is that scalable oversight was never conceived as a single technique but as a genuinely open design space, of which reward-prediction is only the most literal-minded branch, weak aggregate labels and hierarchical delegation being two structurally different ways of making a scarce oversight signal go further.