Imitation Learning
Deep Reinforcement Learning from Human Preferences — inherited
An alternative approach, mentioned in the introduction, of directly cloning demonstrated behavior rather than learning from preferences; noted as inapplicable to behaviors difficult for humans to demonstrate, such as controlling non-human robot morphologies.
If you already know what you want, why not just show it? Cloning a demonstration is often the simplest way to teach a behavior, but not every behavior can be demonstrated by a human body in the first place.
The page introduces the basic method and a couple of its variants, then covers the specific limitation, behaviors humans cannot physically demonstrate, that leads the paper to set this approach aside in favor of comparisons instead.
What imitation learning is
Imitation learning trains a policy directly from demonstrated behavior rather than from any reward signal. In its simplest form, behavioral cloning, a policy (\pi_\theta(a \mid s)) is fit by supervised learning to match the actions a demonstrator took in each observed state, treating the problem as classification or regression over state-action pairs rather than as reinforcement learning. More sophisticated variants, such as generative adversarial imitation learning (Ho and Ermon, 2016), instead train the policy so its trajectories become indistinguishable from the demonstrator's under an adversarially learned discriminator, avoiding some of the compounding-error problems that plague naive cloning. Distinct from imitation learning proper is inverse reinforcement learning (Ng and Russell, 2000), which infers a reward function that would explain the demonstrations and hands that reward to a standard RL algorithm.
Why the paper sets it aside
The paper's introduction names imitation learning, alongside inverse RL, as the natural alternative to preference-based learning whenever demonstrations exist: "If we have demonstrations of the desired task, we can extract a reward function using inverse reinforcement learning... More directly, we can use imitation learning to clone the demonstrated behavior" (deep-rl-human-prefs, §"1 Introduction", p. 1). It then states the limitation motivating the rest of the paper: these approaches "are not directly applicable to behaviors that are difficult for humans to demonstrate (such as controlling a robot with many degrees of freedom but very non-human morphology)" (deep-rl-human-prefs, §"1 Introduction", p. 1). A person can demonstrate scrambling an egg by moving their own arm, but cannot puppeteer a Hopper or Half-Cheetah's joints through a backflip or a one-legged gait; they can only recognize a good one when they see it, exactly the gap comparison-based learning is built to fill.