Cooperative Inverse Reinforcement Learning (CIRL)
Concrete Problems in AI Safety — inherited; Deep Reinforcement Learning from Human Preferences — inherited
A prior framework (Hadfield-Menell et al. 2016) in which an agent and a human collaborate, with the agent inferring and acting on the human's goals.
Cooperative Inverse Reinforcement Learning, or CIRL, is a framework from Hadfield-Menell and colleagues (2016) in which human and agent play on the same team: the agent does not know the goal and must learn it from the human while helping to pursue it.
A shared-payoff game built around a hidden parameter supplies the formalism, put to two distinct jobs for Concrete Problems. The shutdown problem picks up the thread from here, for where that same trust breaks down.
A pre-existing framework for human-agent collaboration
Cooperative Inverse Reinforcement Learning, CIRL, predates Concrete Problems; it is a framework introduced by Hadfield-Menell, Dragan, Carroll, and Russell (2016) for structuring interaction between an agent and a human so the agent works to achieve the human's goals rather than a goal fixed in advance (concrete-problems, §"Multi-Agent Approaches:", p. 6). Concrete Problems inherits it as one concrete instantiation of a broader reframing it calls Multi-agent approaches (to side effects): instead of penalizing environmental change directly, treat side-effect avoidance as a matter of understanding and protecting the interests of the other agents, including humans, that share the environment.
The formalism: a shared-payoff game with a hidden parameter
A CIRL problem is formalized as a two-player Markov game between a human $H$ and a robot $R$ sharing a common state space $\mathcal{S}$, each with its own action set, $A^H$ and $A^R$, and a common transition function. The reward is a single function $R(s, a^H, a^R, \theta)$ parameterized by $\theta \in \Theta$, the human's private goal, which is known to the human but not to the robot. Both players receive the same reward, so the game is one of pure common payoff, not of competition, and the robot's problem reduces to a partially observable planning problem: it maintains a belief $b(\theta)$ over the human's goal and updates that belief via Bayesian inference from the human's observed actions, choosing $a^R_t$ to maximize expected discounted return under its current belief, $\mathbb{E}_{\theta \sim b}\left[\sum_t \gamma^t R(s_t, a^H_t, a^R_t, \theta)\right]$. Because both players are trying to help the human's true objective succeed, an optimal human policy under this game will actively teach, demonstrating or clarifying its goal rather than simply pursuing it, whenever teaching improves the robot's eventual behavior more than direct pursuit would.
What the framework buys the paper, twice over
CIRL does double duty in Concrete Problems. Within Side effects and control as a relationship with other agents, it is paired with Reward autoencoder (goal transparency) as a way of making agent-human interaction safer through mutual legibility, but the two run legibility in opposite directions: CIRL makes the human's goal legible to the agent, while the reward autoencoder tries to make the agent's goal legible to the human. CIRL also supplies a candidate mechanism for the Shutdown / off-switch problem: because the agent infers the human's goal rather than optimizing one fixed in advance, allowing itself to be shut down is simply what a correctly functioning CIRL agent does whenever the human's inferred goal favors shutdown, with no separate off-switch logic required, though the paper still calls shutdown "an interesting problem in its own right" rather than a settled one (concrete-problems, §"Multi-Agent Approaches:", p. 6). Within the second theme it belongs to, Learning from demonstrations instead of a specified reward, CIRL is grouped with Use demonstrations (safe exploration) as a way of letting human intentions substitute for a hand-specified reward, but it is the interactive counterpart of that offline technique: where use-demonstrations mines a fixed, pre-collected dataset of expert trajectories, CIRL sets up an ongoing two-player game in which the agent updates its belief about the human's goal turn by turn, buying it the ability to ask clarifying questions or defer to the human in a way an agent trained once on a fixed dataset cannot.
A 2017 instance: preferences as the only channel
The 2017 paper describes its entire setup as an instance of this framework, not merely an analogy to it. Its related work states plainly: "This framework considers a two-player game between a human and a robot interacting with an environment with the purpose of maximizing the human's reward function. In our setting the human is only allowed to interact with this game by stating their preferences" (deep-rl-human-prefs, §"1.1 Related Work", p. 3). Where the general CIRL game lets the human act in the environment however is most informative, demonstrating, correcting, or teaching by any available channel, the 2017 paper collapses the human's move set down to a single act: choosing which of two trajectory segments is preferred (Cooperative Inverse Reinforcement Learning (CIRL) — is narrowed to a preferences only channel in → Preference Elicitation Protocol). That narrowing is what Preference Elicitation Protocol formalizes, and it is also what makes the 2017 setting tractable at deep-RL scale: teaching by full demonstration or by acting inside the shared game is no longer required, only a comparison judgment is, which is precisely the channel the paper scales up to feedback on less than 1% of the agent's interactions with the environment (deep-rl-human-prefs, §"Abstract", p. 1).