Feudal reinforcement learning
Concrete Problems in AI Safety — inherited
An early hierarchical RL architecture (Dayan & Hinton 1993) in which managerial agents set goals for sub-agents, cited as the origin of hierarchical RL.
In 1993, Dayan and Hinton sketched a control scheme run like a fiefdom: manager agents set goals, and worker agents chase synthetic rewards for meeting them. Feudal reinforcement learning is that early architecture, remembered here as the starting point of hierarchical RL.
The prose covers the manager-worker arrangement and its synthetic rewards first, then what the design lends to scalable oversight and the warning that travels with it.
Managers, workers, and synthetic reward
Feudal reinforcement learning, introduced by Dayan and Hinton in 1993, is an early hierarchical-control architecture in which a "manager" agent operating at a coarse timescale sets a goal for a "worker" agent below it, and the worker is scored not on the manager's real objective but on a synthetic, densely available reward for completing the assigned sub-goal. The architecture can recurse: a manager's worker may itself act as a manager for finer-grained sub-workers, with real, primitive actions only appearing at the bottom of the hierarchy. It predates Concrete Problems by more than two decades and is cited as the concrete lineage behind the paper's account of Hierarchical reinforcement learning (HRL) (concrete-problems, §"5 Scalable Oversight", p. 13).
What it lends to scalable oversight — and a warning that comes with it
The edge Hierarchical reinforcement learning (HRL) — traces its lineage to → Feudal reinforcement learning credits Dayan and Hinton's manager-worker split as the specific delegation structure the paper proposes for Scalable oversight: pushing expensive, sparse evaluation of a true objective up to a small number of abstract, top-level decisions while cheap synthetic reward handles everything below. But the same architecture that solves the oversight problem also reproduces it in miniature. A footnote the paper attaches to this discussion observes that once delegation is implemented, sub-agents may take actions that do not serve the top-level agent's real goals, "in the same way that a human may be concerned that the top-level agent's actions don't serve the human's real goals" (concrete-problems, §"5 Scalable Oversight", p. 13, n. 4) — feudal RL's own structure is flagged as a recursive instance of the alignment problem it was recruited to help solve. The concept is a child of Hierarchical reinforcement learning (HRL) within the Scaling oversight without a learned reward predictor theme, sits in the Two or more remedies for the identical problem, sorted by where they intervene connective theme, and belongs to the Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline supertheme.