Use demonstrations (safe exploration) — is explicitly passed over in favor of → Preference Elicitation Protocol

Concrete Problems (2016) proposes using demonstrations, mined via inverse RL or apprenticeship learning, as a way to substitute a human's expertise for a hand-specified objective (concrete-problems, §"Use Demonstrations:", p. 15). The 2017 paper considers exactly this family of techniques in its own introduction and rejects it for a specific structural reason: inverse reinforcement learning and imitation learning both require a human able to demonstrate the desired behavior, which breaks down for behaviors 'difficult for humans to demonstrate (such as controlling a robot with many degrees of freedom but very non-human morphology)' (deep-rl-human-prefs, §"1 Introduction", p. 1). The preference elicitation protocol is built to survive exactly that failure: it only ever asks a human to compare two rendered clips and say which is better, a tie, or incomparable, never to produce or execute the behavior itself (deep-rl-human-prefs, §"2.2.2 Preference Elicitation", p. 5). A demonstration is a weak superset of a preference in one sense — anyone who can demonstrate a behavior can also judge it — but the reverse fails, which is precisely the asymmetry the 2017 paper exploits: recognition is cheaper and more available than execution.