Qbert (Atari game)
Deep Reinforcement Learning from Human Preferences — inherited
An Atari game used as a standard benchmark task; the paper's method failed to learn to beat the first level using real human feedback, possibly because short video clips of Qbert are confusing and difficult for raters to evaluate.
Most of the paper's Atari results show human feedback trailing a synthetic baseline only modestly. One game is the exception, where real human feedback simply failed to clear the first level.
The page covers the task's objective, then this one clear failure and the paper's own tentative explanation for it, before closing on why that explanation lines up with a finding documented elsewhere in the paper but was never actually tested here.
What the task is
Qbert is one of the seven Atari benchmark games; contractors were instructed to "change the color of as many blocks as you can (good), but don't jump off the side or run into enemies (very bad)" (deep-rl-human-prefs, §"B.2 Atari", p. 17). Under synthetic feedback, it is one of the slower successes in the paper's Atari results: "synthetic feedback eventually performs near the level of RL but learns more slowly" (deep-rl-human-prefs, §"3.1.2 Atari", p. 7).
The one clear failure with real human feedback
Qbert is the paper's explicit counterexample to its own real-human results: "our method fails to learn to beat the first level with real human feedback; this may be because short clips in Qbert can be confusing and difficult to evaluate" (deep-rl-human-prefs, §"3.1.2 Atari", p. 8). Unlike most other games, where human feedback trails synthetic feedback only modestly, Qbert's real-human result does not merely underperform — it fails outright to clear the first level.
A prediction the paper never actually tests
The Clip-Length Effects on Human Evaluation edge notes that this failure lines up exactly with a mechanism documented elsewhere in the paper — that short clips take a rater disproportionately long to interpret relative to the information they convey (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 10) — but that finding was verified only via a length-1 ablation on the MuJoCo robotics tasks, since "our Atari reward model depends on a sequence of consecutive frames rather than a single frame" (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 10, footnote 5). Qbert's difficulty is consistent with the clip-length story but was never directly tested against it.