Learning from demonstrations instead of a specified reward

Members — a guided walk

1. Inverse reinforcement learning (IRL)

   Inverse reinforcement learning infers a reward function or policy from observed expert trajectories rather than from a hand-specified reward, and this walk follows how that basic idea gets weighed against alternatives, repurposed, and eventually built into a real training pipeline. IRL is the foundational technique this theme measures its other demonstration-based proposals against, starting with the most direct alternative to it.

2. Imitation Learning

   Imitation learning skips reward inference entirely and clones the demonstrated behavior directly, the same basic contrast Concrete Problems draws between IRL and cloning in 2016. The 2017 paper revisits this exact pair the following year and gives the stated reason both get passed over there: IRL and imitation learning both require a human able to demonstrate the desired behavior, which breaks down for tasks like controlling a non-human robot morphology a person can recognize but cannot perform.

3. Apprenticeship learning

   Apprenticeship learning returns to IRL's own family of techniques rather than imitation learning's direct-cloning alternative: it borrows its technique from inverse reinforcement learning but targets a different object, letting an agent learn near-optimal behavior directly from expert trajectories, skipping trial-and-error exploration entirely.

4. Use demonstrations (safe exploration)

   Use demonstrations is the proposed safety remedy that repurposes both prior techniques, IRL and apprenticeship learning, as ways to reduce risky exploration: learn a baseline policy from expert trajectories first, then optionally bound any further exploration around it. It is the umbrella under which the two preceding techniques become a safety intervention rather than just a learning method.

5. Cooperative Inverse Reinforcement Learning (CIRL)

   Cooperative inverse reinforcement learning is grouped alongside use demonstrations as its interactive counterpart, sharing the goal of letting a human's intentions substitute for a hand-specified reward but working through ongoing collaboration rather than a fixed, pre-collected dataset of trajectories. It marks the edge of what the demonstration-based proposals in Concrete Problems covered, before the walk turns to how one of them was actually built.

6. Supervised fine-tuning (SFT)

   Supervised fine-tuning is InstructGPT's real-world operationalization of use demonstrations: a pretrained GPT-3 model is fine-tuned directly on human-written demonstrations of desired behavior, turning a 2016 proposal into the concrete first stage of a 2022 training pipeline. It is structurally the simplest of InstructGPT's three steps precisely because it inherits use demonstrations' original simplicity.

7. SFT dataset

   The SFT dataset is the roughly 13,000 labeler-written and API-sourced demonstrations that supervised fine-tuning actually trains on, the concrete substance behind the abstract phrase human demonstrations. Its construction, including duplicated template instructions, shapes what the resulting model learns.

8. Supervised fine-tuned (SFT) model

   The SFT model is what supervised fine-tuning on the SFT dataset produces, and it inherits that dataset's quirks directly: because near-duplicate demonstrations were common early in the project, the model was prone to early overfitting despite being trained for many epochs anyway. It closes this walk as both a standalone baseline and, as later stages of the pipeline depend on, the seed for everything downstream.

Superthemes

• Bounding what an agent can change or explore, not just what it's told to want
• Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline

Edges explored within this theme

• Use demonstrations (safe exploration) — operationalized as → Supervised fine-tuning (SFT) hindsight
• Apprenticeship learning — borrows technique from → Inverse reinforcement learning (IRL)
• Inverse reinforcement learning (IRL) — repurposed for → Use demonstrations (safe exploration)
• Cooperative Inverse Reinforcement Learning (CIRL) — interactive counterpart of → Use demonstrations (safe exploration)
• SFT dataset — duplication underlies early overfitting of → Supervised fine-tuned (SFT) model
• Use demonstrations (safe exploration) — is explicitly passed over in favor of → Preference Elicitation Protocol

Super edges touching this theme

• Side effects and control as a relationship with other agents — passes on only its demonstration half to → Learning from demonstrations instead of a specified reward hindsight
• Keeping exploration inside a known-recoverable region — is the only one of its mechanisms to persist into → Learning from demonstrations instead of a specified reward hindsight
• Learning from demonstrations instead of a specified reward — displaces its exploration bound onto → The mechanics that make PPO fine-tuning work hindsight

Sources

Every passage behind this theme’s member concepts.