Quantitative vs. Qualitative Evaluation Methodology
Deep Reinforcement Learning from Human Preferences — introduced
The paper's two-track evaluation methodology: 'quantitative' evaluation compares agent performance against a known-but-withheld true reward function on standard RL benchmark tasks, while 'qualitative' evaluation, used for tasks with no reward function at all, judges whether the trained agent fulfills a natural-language goal description as assessed by a human watching videos of its behavior.
An agent that never observes its own reward function poses an obvious question: how do you tell whether it succeeded? The paper answers it two different ways, depending on whether a reward function exists to check against at all.
The page lays out both evaluation tracks and what each requires, then closes on how the second track's method, judging behavior against a stated goal rather than a number, already looks like an early sketch of something later systems do at much larger scale.
The paper's two tracks of evaluation
The paper needs a way to judge success under a stated premise — that the agent never observes a reward signal — which makes ordinary reward curves unavailable by definition. It resolves this by defining two parallel evaluation tracks. Quantitative evaluation applies when preferences "are generated by a reward function r," in which case "our agent ought to receive a high total reward according to r" even though r was withheld during training, letting the authors "evaluate the agent quantitatively" against a benchmark task's real score (deep-rl-human-prefs, §"2.1 Setting and Goal", p. 4). Qualitative evaluation applies when "we have no reward function by which we can quantitatively evaluate behavior... the situation where our approach would be practically useful" (deep-rl-human-prefs, §"2.1 Setting and Goal", p. 4). All of the MuJoCo and Atari benchmark results in §3.1 use the quantitative track; the novel-behavior demonstrations in §3.2, such as the Hopper Backflip Demonstration and Enduro Keeping-Pace Demonstration, use the qualitative track exclusively, since no ground-truth reward for a backflip or for keeping pace with traffic exists to check against.
What each track actually requires
The quantitative track requires a task that already has a standard reward function the experimenters can withhold from the agent and labelers but still consult afterward for scoring — every OpenAI Gym benchmark task qualifies. The Synthetic Oracle Feedback device is "only realizable" inside this track, because manufacturing the oracle's answer means consulting exactly that withheld reward function (deep-rl-human-prefs, §"2.1 Setting and Goal", p. 4). The qualitative track instead "start[s] from a goal expressed in natural language, ask[s] a human to evaluate the agent's behavior based on how well it fulfills that goal, and then present[s] videos of agents attempting to fulfill that goal" (deep-rl-human-prefs, §"2.1 Setting and Goal", p. 4) — there is no reward number to report, only a human judgment of whether the goal was met.
An early sketch of instruction following
The qualitative track's procedure — state a goal in language, have a human judge whether behavior fulfills it, using video rather than a scalar reward — is already the shape instruction following takes five years later. The Instruction following (as a fine-tuning objective) edge (hindsight) observes that this is the paper's "only recourse" for tasks with no reward function at all, and that shape carries forward into how InstructGPT's labelers judge outputs against a stated instruction rather than against any hand-written scoring function. The 2017 paper never uses the word "instruction," but its qualitative methodology is functionally the same move: judging behavior against a stated goal rather than against a number.