Trajectory Segment
Deep Reinforcement Learning from Human Preferences — introduced
A short fixed-length sequence of observation-action pairs sampled from an agent's trajectory (1-2 seconds of behavior in this paper's experiments), used as the basic unit shown to and compared by a human, rather than whole trajectories or single states.
Asking a human to judge a whole episode, or a single frame, both have problems: one is too much to hold in mind, the other too little to say anything about. A short clip somewhere in between is the unit this method actually settles on.
The page covers what a segment is and how it's shown to a rater, then how it feeds directly into the comparison math, before closing on why the specific one-to-two-second length was chosen and how the same underlying idea, comparing short units rather than whole trajectories, carries over once the domain shifts to text.
The unit a human actually judges
A trajectory segment is defined formally as $\sigma = ((o_0,a_0),(o_1,a_1),\ldots,(o_{k-1},a_{k-1})) \in (O\times A)^k$, a short slice of an agent's observation-action history (deep-rl-human-prefs, §"2.1 Setting and Goal", p. 3). In the paper's experiments a segment is rendered as a short video clip between 1 and 2 seconds long (deep-rl-human-prefs, §"2.2.2 Preference Elicitation", p. 5). This is a deliberate departure from earlier preference-based RL: Akrour et al. (2012, 2014) elicited preferences over whole trajectories rather than short clips (deep-rl-human-prefs, §"1.1 Related Work", p. 3), and Wilson et al. (2012) could reset their environment to an arbitrary state before generating a trajectory to compare. The paper's high-dimensional environments cannot be reset that way, so the two segments in a pair generally begin from different states — a complication the algorithm has to tolerate, since the human sees no indication of where in an episode either clip was drawn from.
Feeding the comparison protocol and the loss
The segment is the object every other piece of the method operates on. The Preference Elicitation Protocol never shows the human anything larger or smaller than a pair of segments — not a whole episode, not a single frame (deep-rl-human-prefs, §"2.2.2 Preference Elicitation", p. 5). And when the Bradley-Terry Model converts a segment into a scalar for its softmax, it sums predicted reward $\hat r(o_t,a_t)$ over every step of the segment with no discount factor applied to later timesteps (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). The paper flags this as a modeling choice rather than a neutral default: undiscounted summation can be read as assuming the human is indifferent to when within the clip good or bad things happen, and discounting or inferring the human's own discount function would also have been reasonable.
Why one to two seconds
The segment length is not an arbitrary round number. Shorter clips took raters disproportionately longer to understand relative to the information they conveyed, so the authors chose the shortest length at which evaluation time still scaled roughly linearly with duration; single frames required substantially more comparisons to reach the performance clips achieved (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 10). This finding, tracked in the Clip-Length Effects on Human Evaluation concept, is what calibrates the 1-2 second figure above; it is an output of the ablations, not an input to them.
The unit that generalizes
Comparing short clips of behavior is specific to sequential control tasks with a rendered environment. When the same two-stage recipe — learn a reward from comparisons, then optimize it — was carried into language models, the atomic unit of comparison changed along with the domain: InstructGPT's RM (comparison) dataset has labelers rank complete text completions to the same prompt rather than segments of an ongoing trajectory. The underlying commitment to comparisons over absolute scores survives the transition; what counts as the thing being compared does not.