Synthetic Oracle Feedback
Deep Reinforcement Learning from Human Preferences — introduced
An experimental device in which a synthetic 'oracle' answers preference queries according to the ground-truth task reward (indicating whichever segment actually receives higher true reward), used to isolate the effect of the reward-learning method from the noise and cost of real human raters.
Checking a method that works without an observable reward against the truth requires access to that same reward somewhere, just not where the agent or the labelers can see it. A synthetic stand-in for a human answers exactly this need.
The page covers how the oracle answers and where it stays silent, then closes on a surprising shaping advantage that shows up even with no human anywhere in the loop, complicating a simpler story about human judgment being what mattered.
Why the paper needs a synthetic oracle
The paper's whole method is built to work when no reward function is observable, but demonstrating that convincingly requires a way to check the method against ground truth without cheating. The synthetic oracle supplies exactly that check: rather than sending a query to a human, the experiment "immediately repl[ies] by indicating a preference for whichever trajectory segment actually receives a higher reward in the underlying task" (deep-rl-human-prefs, §"3.1 Reinforcement Learning Tasks with Unobserved Rewards", p. 6). This isolates the effect of the reward-learning algorithm itself from the noise, inconsistency, and expense of a real rater, letting Figures 2 and 3 compare human feedback, oracle feedback, and direct RL on the true reward side by side. As the Quantitative vs. Qualitative Evaluation Methodology edge notes, the oracle only makes sense on the quantitative half of the paper's evaluation methodology, since manufacturing its answer requires a withheld ground-truth reward function that the qualitative tasks (backflips, one-legged running) deliberately lack (deep-rl-human-prefs, §"2.1 Setting and Goal", p. 4).
How the oracle answers, and where it goes silent
The oracle's rule is mechanical: compare the true total reward summed over each segment and prefer whichever is higher. When both segments earn identical true reward, which the paper notes is "relatively common" in sparse-reward Atari games, the oracle outputs indifference rather than a forced choice; the paper reports that "such ties never made up a large majority of our data" while still carrying "significant information to the reward predictor as long as they are not too common" (deep-rl-human-prefs, §"3.1 Reinforcement Learning Tasks with Unobserved Rewards", p. 6, footnote 4). The Environment Modifications for Preference Learning edge marks one place where oracle feedback is treated differently from the general rule: the standard Atari modification strips life-loss and episode-end signals from every game so termination itself cannot leak task information, but "when providing synthetic oracle feedback we replace episode ends with a penalty in all games except Pong" (deep-rl-human-prefs, §"A Experimental Details", p. 14) — Pong (Atari game) alone keeps its natural termination under the oracle.
A non-human instance of reward shaping
The paper's other headline shaping result, illustrated by Ant (MuJoCo task), is usually told as a story about human judgment specifically. The oracle results complicate that story: at 1400 labels, an agent trained against the synthetic oracle's comparisons "performs slightly better than if it had simply been given the true reward, perhaps because the learned reward function is slightly better shaped" (deep-rl-human-prefs, §"3.1.1 Simulated Robotics", p. 7). The same shaping advantage attributed elsewhere to human preferences also shows up with no human in the loop, via the Human-Feedback Implicit Reward Shaping edge — the reward-model architecture itself, not only human judgment, contributes to the improved shaping.