Reward Predictor Ensemble — supplies the disagreement signal for → Uncertainty-Based Query Selection (Ensemble Variance)

Uncertainty-based query selection needs some proxy for how uncertain the reward predictor is about a candidate pair, and the reward predictor ensemble is where that proxy comes from: with three independently bootstrap-trained predictors, the query-selection procedure samples a large batch of candidate segment pairs, asks every ensemble member for its preference prediction, and keeps whichever pairs show the highest variance across members (deep-rl-human-prefs, §"2.2.4 Selecting Queries", p. 6). The coupling between the two is tight enough that removing the ensemble removes the query-selection method entirely: the paper's 'no ensemble' ablation trains a single predictor and, having no members left to disagree, falls back to picking queries uniformly at random (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 9). The authors are explicit that ensemble variance is only 'a crude approximation' to true uncertainty, distinguishing it from a more principled expected-value-of-information criterion they leave to future work. So the ensemble's disagreement is not a side benefit of using multiple predictors for accuracy; in this paper it is the only uncertainty signal the query-selection method has access to.