Uncertainty-Based Query Selection (Ensemble Variance)
Deep Reinforcement Learning from Human Preferences — introduced
The specific query-selection heuristic used: sample a large batch of candidate segment pairs, have every member of the reward-predictor ensemble predict a preference for each, and present to the human those pairs whose predicted preference has the highest variance across ensemble members.
The paper's rule for choosing which pairs of clips to show a human leans on something simple: how much its own ensemble of predictors disagrees about them.
The page covers how the heuristic samples and filters candidate pairs, then the authors' own admission that it is a crude approximation to a harder problem, before closing on how it counts as an early, working version of an idea named as an open research direction a year earlier.
Picking which pairs to ask about
Not every candidate pair of segments is equally useful to show a human. The paper's heuristic samples a large batch of candidate segment pairs of length $k$, has every member of the Reward Predictor Ensemble predict which segment will be preferred in each pair, and selects for the human those pairs whose predicted preference has the highest variance across ensemble members (deep-rl-human-prefs, §"2.2.4 Selecting Queries", p. 6). The default setup draws ten candidate pairs for every one ultimately shown to the human (deep-rl-human-prefs, §"A Experimental Details", p. 14) — a way of economizing scarce human attention that anchors the The economics of human feedback theme.
A crude approximation, by the authors' own account
The paper does not present this as a considered solution. It calls ensemble-variance querying "a crude approximation" to the query-selection problem, and its own ablations show that in some tasks it actually impairs performance relative to picking queries at random (deep-rl-human-prefs, §"2.2.4 Selecting Queries", p. 6; §"3.3 Ablation Studies", p. 9). The more principled criterion, expected value of information, credited to Akrour et al. (2012) and Krueger et al. (2016), is named explicitly and set aside as future work rather than implemented (deep-rl-human-prefs, §"2.2.4 Selecting Queries", p. 6).
An early instance of a named research direction
The heuristic is also notable for what it instantiates. Concrete Problems (2016) had named semi-supervised or active reward learning as an open research direction: combine reward prediction with active learning so an agent can request feedback specifically on the events most worth asking about (concrete-problems, §"5 Scalable Oversight", p. 12). This paper, co-authored by two of the same researchers, gives that proposal one of its first concrete implementations in deep RL, even while describing its own version of the idea as crude.