Expected-Value-of-Information Query Selection
Deep Reinforcement Learning from Human Preferences — inherited
A more principled query-selection criterion, mentioned as future work, that would choose queries by their expected value of information rather than by ensemble disagreement, attributed to Akrour et al. (2012) and Krueger et al. (2016).
Some questions are more worth asking than others, and picking the right one is its own unsolved problem. This is the more principled way of choosing, one the paper names but never actually builds.
The page explains what scoring a query by its expected value of information would mean, then how the paper's own simpler heuristic falls short of it by its authors' own admission, and closes on why this matters for the economics of a limited budget of human comparisons.
What expected value of information means here
Expected-value-of-information (EVOI) query selection chooses the next comparison to ask a human by estimating how much each candidate query would be expected to reduce uncertainty, or downstream regret, in the learned reward function or resulting policy, rather than by a proxy for uncertainty alone. Concretely, this means scoring a candidate query by something like the expected reduction in loss the human's answer would produce, averaged over the possible answers weighted by their currently predicted probability, and then asking whichever candidate maximizes that quantity. This is a more direct optimization of "which question is worth asking" than measuring how much the model's current predictors merely disagree with each other, since disagreement does not necessarily track how much a query would change the resulting behavior.
Its status: cited, not built
The paper is explicit that its own query-selection method, Uncertainty-Based Query Selection (Ensemble Variance) by ensemble variance, is "a crude approximation" to this harder problem (deep-rl-human-prefs, §"2.2.4 Selecting Queries", p. 6). It names EVOI querying, attributed to Akrour et al. (2012) and Krueger et al. (2016), as what a more principled criterion would look like, in the same breath as noting that ensemble disagreement "actually impairs performance" on some tasks in its own ablations (deep-rl-human-prefs, §"2.2.4 Selecting Queries", p. 6). The paper leaves EVOI entirely to future work rather than implementing or testing it (deep-rl-human-prefs, §"2.2.4 Selecting Queries", p. 6).
Why it belongs to the economics of feedback
The point of any query-selection rule is to make each of the roughly 700 to 5,500 human comparisons the paper spends on a task count for as much learning as possible, since comparisons, unlike agent-environment interactions, cost human time. EVOI query selection targets that economics problem directly, asking the query expected to matter most rather than the query the ensemble merely disagrees about, which is why it sits under Semi-supervised or active reward learning as the more principled endpoint the paper's own heuristic only approximates.