Inverse reinforcement learning (IRL)
Concrete Problems in AI Safety — inherited; Deep Reinforcement Learning from Human Preferences — inherited
A prior family of methods that infers a reward function (or policy) from observed expert trajectories rather than from a hand-specified reward.
Rather than being handed a reward function, an agent can watch an expert and work backward to the reward that would explain the behavior. That inversion, formalized by Ng and Russell in 2000, is inverse reinforcement learning.
The recovery idea gets explained first, followed by the two distinct safety jobs the corpus finds for it. Apprenticeship learning and using demonstrations for safe exploration are both worth following, each putting the same inversion to different use.
Recovering a reward function from behavior
Inverse reinforcement learning (IRL), formalized by Ng and Russell in 2000, inverts the usual reinforcement-learning problem: instead of computing an optimal policy from a given reward function, it infers a reward function (or a policy derived from one) that best explains a set of expert demonstrations. Given an MDP without its reward function and a set of trajectories generated by a near-optimal expert policy $\pi^*$, classical IRL seeks a reward $R$ under which $\pi^*$ is optimal or near-optimal — formally, a reward such that for every state-action pair, the expert's action-value dominates the value of any alternative action, $Q^{\pi^*}(s,a^*) \geq Q^{\pi^*}(s,a)$ for all $a$. The method predates Concrete Problems by over fifteen years and had, by the time the paper was written, already been extended to use deep neural networks to learn the cost function or policy directly from demonstrations (concrete-problems, §"Reward Pretraining:", p. 10; §"Use Demonstrations:", p. 15).
Two safety jobs for one technique
Concrete Problems cites IRL twice, for structurally different purposes, which is what gives this concept unusual reach across the corpus. Under reward pretraining, IRL is proposed as a defense against reward hacking: train a fixed reward function from demonstrations before any interaction with the environment begins, so the agent has no opportunity to manipulate its own reward signal later. Under Use demonstrations (safe exploration), the identical technique is repurposed to address the different problem of unsafe exploration — the edge Inverse reinforcement learning (IRL) — repurposed for → Use demonstrations (safe exploration) traces this shift, where the resulting trajectories build a baseline policy that lets an agent skip risky trial-and-error learning or bound further exploration around it, and the edge belongs to the The same technique, cited or deployed twice, doing different work each time connective theme. A second edge, Apprenticeship learning — borrows technique from → Inverse reinforcement learning (IRL), distinguishes IRL from Apprenticeship learning: apprenticeship learning uses an IRL-style optimization only as an internal subroutine toward matching the expert's long-run feature expectations, and never commits to an inspectable reward function the way classical IRL does — a difference the corpus flags as safety-relevant, since only an explicit reward can be checked for hacking vulnerabilities after the fact. The concept sits in the Learning from demonstrations instead of a specified reward theme and belongs to both the Bounding what an agent can change or explore, not just what it's told to want and Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline superthemes; it is also the technical ancestor of the demonstration-learning wager that resurfaces, six years later, as InstructGPT's Supervised fine-tuning (SFT) stage.
The alternative the 2017 paper sets aside
The 2017 paper considers IRL the natural alternative to its own approach and explicitly declines it. Its introduction states: "If we have demonstrations of the desired task, we can extract a reward function using inverse reinforcement learning (Ng and Russell, 2000). This reward function can then be used to train an agent with reinforcement learning" but adds that "these approaches are not directly applicable to behaviors that are difficult for humans to demonstrate (such as controlling a robot with many degrees of freedom but very non-human morphology)" (deep-rl-human-prefs, §"1 Introduction", p. 1). Learning from preference comparisons rather than demonstrations is the paper's answer to exactly this gap: comparisons only require a human to recognize which of two outcomes is better, not to perform the desired behavior, which is why the paper's own list of desiderata opens with solving "tasks for which we can only recognize the desired behavior, but not necessarily demonstrate it" (deep-rl-human-prefs, §"1 Introduction", p. 2). IRL is not refuted here, only set aside for a class of behaviors it cannot reach; the paper's own CIRL framing, narrowed to a preferences-only channel, is the mechanism it substitutes in IRL's place.