Hierarchical reinforcement learning (HRL)
Concrete Problems in AI Safety — inherited
A prior RL paradigm in which a top-level agent takes abstract, long-timescale actions delegated to sub-agents that pursue synthetic, denser reward signals down to primitive actions.
Some agents work like organizations: a top level decides broad, long-horizon moves, and lower levels handle the details in exchange for rewards of their own. Hierarchical reinforcement learning is that longstanding paradigm, which Concrete Problems in AI Safety (2016) revisits because layered delegation might let sparse oversight stretch further.
The page traces that delegation across timescales first, then turns to why the structure promises to stretch scarce oversight further, and to the earlier architecture, feudal reinforcement learning, the idea descends from.
Delegation across timescales
Hierarchical reinforcement learning (HRL) is a longstanding RL paradigm, surveyed extensively before this paper (concrete-problems cites Barto and Mahadevan's review), in which a top-level agent takes a small number of highly abstract actions extending over long temporal or spatial scales and delegates the details of carrying them out to sub-agents. Each sub-agent is incentivized with a synthetic reward representing correct completion of the delegated goal rather than the top-level agent's true objective, and this delegation can recurse through multiple levels before bottoming out in primitive actions on the environment (concrete-problems, §"5 Scalable Oversight", p. 13). The paradigm traces its concrete architectural lineage to Feudal reinforcement learning, Dayan and Hinton's 1993 manager-worker formulation, and by the time Concrete Problems was written had already been explored in combination with neural-network function approximators.
Why it promises scalable oversight
The edge Scalable oversight — is addressed by → Hierarchical reinforcement learning (HRL) explains the mechanical trick that makes HRL relevant to Scalable oversight: because the top-level agent needs the expensive true-objective signal only for a small number of long-horizon decisions, and every sub-agent below it runs entirely on cheap, dense, synthetic reward, the scarce resource — calls to a costly true objective — is spent exactly where the decision space is smallest. Concrete Problems calls this "a particularly promising approach to oversight" (concrete-problems, §"5 Scalable Oversight", p. 13). Set against Distant supervision in Distant supervision — offers a structurally different route than → Hierarchical reinforcement learning (HRL), HRL is shown to weaken the oversight requirement along a different axis than its counterpart: distant supervision keeps a fine per-decision timescale but accepts weaker labels, while HRL keeps a strong, verifiable signal but only requires it rarely. The concept is a child of Scalable oversight and parent of Feudal reinforcement learning, anchors the Scaling oversight without a learned reward predictor theme, sits in the Two or more remedies for the identical problem, sorted by where they intervene connective theme, and belongs to the Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline supertheme as one of the paper's two non-reward-prediction routes to scaling oversight.