Reward Predictor Ensemble
Deep Reinforcement Learning from Human Preferences — introduced
The reward model is implemented as an ensemble of independently-trained predictors (3 by default), each fit on a bootstrap resample of the preference database drawn with replacement, with the final reward estimate given by averaging the independently-normalized outputs of each member.
One trained model can be confidently wrong; several independently trained copies disagreeing with each other is a far more useful signal. That disagreement is exactly what this ensemble is built to produce.
The page covers how the three predictors are trained and combined, then how their disagreement gets reused to decide which comparisons are worth asking a human about, and closes on how the ensemble sits alongside the paper's other ways of keeping the reward model well-behaved.
Three predictors, bootstrap-trained
The reward model is not a single network but an ensemble, three predictors by default, each fit on $|D|$ triples resampled from the preference database $D$ with replacement (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). The final estimate $\hat r$ averages the ensemble members' outputs after independently normalizing each one (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5), so no single member's arbitrary scale distorts the aggregate. This bootstrap-and-average structure is standard practice for estimating a model's uncertainty from disagreement among differently-trained copies of itself.
Turning disagreement into a query signal
The ensemble's members are used for more than averaging. Because each is trained on an independent resample, they can each predict a preference for the same candidate pair of segments, and how much they disagree becomes a proxy for how uncertain the reward model is about that pair. This is exactly what Uncertainty-Based Query Selection (Ensemble Variance) consumes: it samples a large batch of candidate pairs, asks every ensemble member to predict a preference for each, and keeps whichever pairs show the highest variance across members to send to the human (deep-rl-human-prefs, §"2.2.4 Selecting Queries", p. 6). Without the ensemble there is no variance to measure; the "no ensemble" ablation trains a single predictor and falls back to picking queries uniformly at random.
Alongside the model's other stabilizers
The ensemble is one of several sibling mechanisms under Reward model (RM) that keep the predictor well-behaved outside its training distribution: Adaptive L2 Regularization for the Reward Predictor bounds its generalization gap, and Reward Prediction Normalization fixes its otherwise-arbitrary output scale. The ensemble's own contribution differs from either — it supplies a measure of epistemic uncertainty that neither of the other two mechanisms provides.