Hierarchical reinforcement learning (HRL) — traces its lineage to → Feudal reinforcement learning

Feudal reinforcement learning (Dayan & Hinton, 1993) is the concrete architecture the paper credits for the delegation structure it proposes for scalable oversight: a manager sets a goal for a subordinate, who is scored on a synthetic reward for completing it rather than on the manager's real objective. Citing this lineage lets the paper make an observation that goes beyond the architecture itself — a footnote notes that when this delegation is implemented, sub-agents may take actions that don't serve the top-level agent's real goals, 'in the same way that a human may be concerned that the top-level agent's actions don't serve the human's real goals' (concrete-problems, §"5 Scalable Oversight", p. 13, n. 4). Feudal RL's manager-worker split is thus flagged as a miniature, recursive instance of the very alignment problem hierarchical RL was proposed to help solve.