Clip-Length Effects on Human Evaluation
Deep Reinforcement Learning from Human Preferences — introduced
An empirical finding that comparing longer video clips is more informative per clip (though less informative per frame) than comparing single frames, because short clips take raters longer to understand relative to the information they convey; the paper chooses the shortest clip length for which rater evaluation time scales roughly linearly with length.
Showing a rater a single video frame sounds like it should be quicker to judge than a longer clip, but it isn't necessarily faster: a short clip can take just as long to make sense of as a long one, for none of the right reasons.
The page walks through what the ablation found about clip length and rater time, then what it calibrated elsewhere in the paper, and closes on the one Atari game where a labeling failure looks like this same effect but was never directly tested against it.
What the ablation found
This finding surfaces from the ablation studies, not from the main method description: the authors "discovered that asking humans to compare longer clips was significantly more helpful per clip, and significantly less helpful per frame. We found that for short clips it took human raters a while just to understand the situation, while for longer clips the evaluation time was a roughly linear function of the clip length" (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 10). The design rule that follows is practical rather than theoretical: "we tried to choose the shortest clip length for which the evaluation time was linear" (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 10) — pick the shortest clip where a rater's time cost stops being dominated by figuring out what's happening.
What it calibrates
The Trajectory Segment concept's headline length spec is a direct output of this finding rather than an arbitrary round number. Clips shown to raters in the main experiments are "between 1 and 2 seconds long" (deep-rl-human-prefs, §"2.2.2 Preference Elicitation", p. 5); the appendix fixes this per domain, comparing "trajectory segments that last 1.5 seconds" in MuJoCo tasks (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14) and segments of "25 timesteps (1.7 seconds at 15 fps with frame skipping)" in Atari (deep-rl-human-prefs, §"A.2 Atari", p. 15). The finding was tested via a "no segments" (length-1) ablation, but only on the robotics tasks, "because our Atari reward model depends on a sequence of consecutive frames rather than a single frame" (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 10, footnote 5) — so the underlying claim about clip length is empirically verified on only half the paper's domains.
A prediction left untested on Qbert
Qbert (Atari game) is the one place in the paper where a labeling failure is explicitly attributed to clip difficulty — "our method fails to learn to beat the first level with real human feedback; this may be because short clips in Qbert can be confusing and difficult to evaluate" (deep-rl-human-prefs, §"3.1.2 Atari", p. 8) — which is exactly the mechanism this finding describes. But because the length-1 ablation never ran on Atari at all, the paper never actually tests whether longer Qbert clips would have helped; the connection is suggestive, not demonstrated.