Apprenticeship learning
Concrete Problems in AI Safety — inherited
A prior learning paradigm (Abbeel & Ng 2005) in which an agent learns near-optimal behavior directly from expert trajectories rather than through trial-and-error exploration.
Abbeel and Ng showed in 2005 that an agent can reach near-optimal behavior straight from expert trajectories, no trial and error required. Apprenticeship learning is their name for that paradigm, and Concrete Problems in AI Safety (2016) leans on it when arguing that demonstrations can substitute for risky exploration.
From there, the prose covers the technique's history before the paper, its mechanism of matching feature expectations, and why the paper cares about the distinction it draws.
A technique that predates the paper
Apprenticeship learning is not a Concrete Problems invention; it is a learning paradigm introduced by Abbeel and Ng (2005) for having an agent acquire near-optimal behavior directly from expert trajectories rather than through trial-and-error exploration, and the paper cites it, alongside inverse reinforcement learning, as one of two named techniques for implementing the "use demonstrations" remedy to unsafe exploration (concrete-problems, §"Use Demonstrations:", p. 15).
The mechanism: matching feature expectations
Given a set of feature functions $\phi: \mathcal{S} \to \mathbb{R}^k$ describing relevant properties of a state, define the feature expectation of a policy $\pi$ as the discounted sum $$\mu(\pi) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t \phi(s_t) \,\middle|\, \pi\right].$$ For any reward function that is a linear combination of these features, $R(s) = w^\top \phi(s)$, the value of a policy is linear in its feature expectations: $V^\pi = w^\top \mu(\pi)$. Apprenticeship learning does not try to recover $w$ or the expert's exact policy; it searches for a policy $\hat\pi$ whose feature expectations are close to those of the expert's demonstrated trajectories, $\|\mu(\hat\pi) - \mu_E\|_2 \le \epsilon$, which guarantees that $\hat\pi$ performs at least as well as the expert under any reward function of that linear form, whatever $w$ actually is. The search typically proceeds by iteratively solving an inverse-RL-style optimization, finding the $w$ that most separates the expert's feature expectations from those of the current best policy, then adding that policy to a growing set and repeating, a max-margin procedure that treats IRL purely as a subroutine toward behavioral matching rather than as an end in itself.
Why the paper cares about the distinction
Apprenticeship learning sits as a child of Use demonstrations (safe exploration) in the paper's safe-exploration proposals, part of the "learn from experts instead of exploring" branch belonging to the theme Learning from demonstrations instead of a specified reward. The concept's edge to Inverse reinforcement learning (IRL) marks a safety-relevant distinction the paper does not spell out itself: because apprenticeship learning outputs a policy rather than an explicit reward function, it cannot be inspected for the same kind of reward-hacking vulnerabilities that a recovered reward function could be checked against, a tradeoff between behavioral guarantees and interpretability that recurs throughout the demonstration-based remedies. As with its sibling concept Use demonstrations (safe exploration), the fact that a baseline learned once from data can substitute for further risky exploration is the property Concrete Problems needs from it, and it is this same substitution, learn from demonstrations instead of specifying or exploring, that InstructGPT's supervised fine-tuning stage later reuses in a simplified, non-IRL form.