Prior human-in-the-loop RL and why it didn't scale — finds its missing piece in → Comparisons as the atomic unit of human feedback

Two of the four prior approaches this theme surveys already tried something close to comparisons, and the gap between their versions and this paper's is exactly what the comparisons theme supplies, a gap neither narrative closes on its own. Akrour et al. (2012, 2014) already elicited preferences rather than absolute reward, but only over whole trajectories and only where reward could be assumed linear in a small set of hand-coded features, "continuous domains with four degrees of freedom and small discrete domains" (deep-rl-human-prefs, §"1.1 Related Work", p. 2); the trajectory-segment format and a deep, nonlinear reward model are what let comparisons scale to Atari's raw pixels and MuJoCo's dozens of degrees of freedom instead. MacGlashan et al. and Pilarski et al. gathered real human feedback too, but "learning only occurs during episodes where the human trainer provides feedback," which the paper calls infeasible for domains needing thousands of hours of experience (deep-rl-human-prefs, §"1.1 Related Work", p. 3). The comparisons theme's asynchronous reward-fitting loop is what decouples labeling from continuous real-time presence, letting a human's few seconds of judgment stand in for supervision the agent keeps using long after the human has looked away. The prior work, in other words, had already located the right question, preferences over demonstrated or observed behavior, without a format cheap and expressive enough to survive contact with deep RL; that format is precisely what this theme adds.