Prior human-in-the-loop RL and why it didn't scale
Learning from something other than a hand-specified reward was not new in 2017; the paper needed a version able to survive deep RL's scale, and none of the existing ones did. Inverse reinforcement learning and imitation learning both need a human able to demonstrate the target behavior, which fails for a robot body no person can inhabit; TAMER's real-time feedback works only for policies learnable in a few hours, not the thousands the paper's Atari tasks demand.
The walk runs through each approach's limits before landing on cooperative inverse reinforcement learning, the framework the paper's method turns out to be one instance of.
Learning a task from something other than a hand-specified reward did not start in 2017, and this theme is the lineage the paper places itself against in its own related-work discussion. Inverse reinforcement learning extracts a reward function from demonstrations of the desired task, an approach the paper notes is inapplicable when the desired behavior is hard for a human to demonstrate in the first place, such as controlling a non-human robot morphology. Imitation learning sidesteps reward learning entirely by cloning demonstrated behavior directly, and fails for the same reason: some behaviors cannot be demonstrated even if they can be recognized. The TAMER framework learns a reward function from real-time human feedback, but the paper distinguishes its own setting by scale: TAMER's prior experiments involve policies learnable relatively quickly, not the thousands of hours of experience the paper's Atari tasks require. Cooperative inverse reinforcement learning frames a human and an agent as two players cooperating to maximize the human's reward function, and the paper positions its own method as a specific instance of this framework, one in which the human's only available move is to state a preference between two segments. None of the four approaches, on their own, could absorb the deep-RL scale the 2017 paper needed: they required either expert demonstrations, hand-coded features, or an amount of continuous human attention that does not fit a budget of minutes.