Use demonstrations (safe exploration)
Concrete Problems in AI Safety — introduced
A proposed remedy that reduces the need for risky exploration by learning from expert demonstration trajectories via inverse RL or apprenticeship learning, optionally bounding further exploration around the resulting baseline policy.
Trial and error is not the only way to learn a job someone competent has already done. Using demonstrations means handing the agent expert trajectories to learn from directly, so risky exploration may not be needed at all; Concrete Problems in AI Safety (2016) proposes it as a remedy for safe exploration.
What follows moves from learning from experts instead of exploring, to the two different jobs one technique performs, to a note that a later paper actually built this one.
Avoiding exploration risk by learning from experts instead
Use demonstrations is a remedy Concrete Problems proposes for Safe exploration: rather than requiring an agent to explore its way to good behavior, provide it with expert trajectories of near-optimal behavior and let it learn from those directly, potentially avoiding the need for risky exploration altogether (concrete-problems, §"Use Demonstrations:", pp. 14-15). The paper names two techniques for doing this, inverse reinforcement learning and apprenticeship learning, and notes that recent progress using deep neural networks to learn a cost function or policy from demonstrations suggested the approach could scale to more advanced RL systems. Even where further learning is necessary after an initial baseline is learned, the paper suggests exploration away from that baseline policy can be bounded in magnitude, feeding directly into Bounded exploration.
One technique, two different jobs
The paper cites inverse reinforcement learning twice for two structurally different purposes, and use-demonstrations is only one of them. Under reward pretraining, IRL defends against reward hacking: a fixed reward function is trained from demonstrations before any interaction with the environment, so the agent cannot later manipulate its own reward signal. Under use demonstrations, the same technique is repurposed for a different problem entirely, unsafe exploration: the same trajectories are used not to lock down a reward function but to build a baseline policy that lets the agent skip risky trial-and-error learning. The tool is identical; what changes is which failure mode the resulting policy or reward is meant to guard against. CIRL sits nearby as an interactive counterpart: where use-demonstrations mines a fixed, pre-collected dataset offline, CIRL runs an ongoing two-player game in which the agent updates its belief about the human's goal turn by turn.
The one technique in this batch that a later paper actually built
Use-demonstrations carries the batch's clearest forward-looking edge. Six years later, InstructGPT's supervised fine-tuning stage operationalizes almost exactly what this concept proposes, but by the simplest mechanism the 2016 paper considered: instead of first recovering a reward function or matching feature expectations the way IRL and apprenticeship learning do, SFT fine-tunes GPT-3 directly on roughly 13,000 human-written demonstrations using ordinary supervised learning, with no reward inference or policy-matching machinery involved (instructgpt, §"3.1 High-level methodology", p. 6). The exploration-safety motivation drops out entirely in the process: SFT is not framed as bounding a later RL exploration phase but as a free-standing baseline and initializer for the stages that follow it. What survives across six years and a change of domain, from robots exploring physical environments to a language model being fine-tuned on text, is the core wager: that a policy learned from demonstrations can substitute for a hand-specified objective. This is why use-demonstrations anchors the theme Learning from demonstrations instead of a specified reward, the corpus's most direct genealogical link between a 2016 proposal and a piece of load-bearing 2022 infrastructure, in addition to its role in Keeping exploration inside a known-recoverable region as the concept whose baseline policy anchors bounded exploration.