Preference Elicitation Protocol — avoids the overfitting problem solved by → K-choose-2 single-batch comparison training

The 2017 protocol only ever asks about one pair at a time: a human sees two clips and returns a single preference, tie, or incomparable judgment, stored as one triple in the preference database with no reference to any other comparison (deep-rl-human-prefs, §"2.2.2 Preference Elicitation", p. 5). Because every training example is an independent pairwise draw, the 2017 paper never faces a problem InstructGPT has to solve directly: what to do when several comparisons all come from the same underlying judgment. InstructGPT collects a full ranking over K = 4 to 9 outputs per prompt in one sitting, which decomposes into C(K,2) pairwise comparisons that are correlated with each other rather than independent; naively shuffling them into ordinary training batches caused the reward model to overfit, so the paper instead places all comparisons from one ranking into a single batch element (instructgpt, §"3.5 Models", p. 8). The k-choose-2 batching trick, in other words, is a fix for a problem the 2017 protocol's strictly one-pair-at-a-time design structurally could not have — a cost of collecting richer, K-wise preference data that pairwise-only elicitation never had to pay.