Elo Score (helpfulness/harmlessness)
Constitutional AI: Harmlessness from AI Feedback — inherited; Deep Reinforcement Learning from Human Preferences — inherited
An explicit analogy to chess's Elo rating system, invoked by the 2017 paper to explain why a Bradley-Terry loss can be fit to differences in predicted reward, without the paper ever computing an Elo rating itself. Constitutional AI turns the analogy into a literal computed metric in 2022, tracking the relative helpfulness and harmlessness of model snapshots from crowdworker pairwise comparisons.
A rating system built for ranking chess players from wins and losses turns out to work for ranking model responses too, but only one of the two papers here actually computes it; the other just borrows the idea to explain a formula.
The page covers the 2017 paper's use of Elo as a pure explanatory analogy for its comparison-based loss, with no rating ever computed, then how Constitutional AI (2022) uses it literally, five years later, to track helpfulness and harmlessness across training snapshots, plus a caveat about the instrument's own not-fully-neutral instructions.
An analogy first, in 2017
The Elo rating system enters this corpus two years before Constitutional AI computes anything with it. The 2017 paper never ranks players or models; it invokes Elo purely as an explanatory analogy for why its Bradley-Terry fitting procedure makes sense, understanding the pairwise-comparison loss as "equating rewards with a preference ranking scale analogous to the famous Elo ranking system developed for chess (Elo, 1978). Just as the difference in Elo points of two chess players estimates the probability of one player defeating the other in a game of chess, the difference in predicted reward of two trajectory segments estimates the probability that one is chosen over the other by the human" (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). This is an explanatory device, not a measurement: the paper fits its reward predictor via a Bradley-Terry softmax over summed rewards, and Elo supplies the intuition for why a difference in an unbounded, uncalibrated scalar can be read as a preference probability, but no Elo rating of any player, model, or policy is computed anywhere in the paper (Elo Score (helpfulness/harmlessness) — supplies the explanatory analogy for → Bradley-Terry Model). The literal, computed Elo rating this concept is named for belongs to Constitutional AI, five years later.
What an Elo rating is
The Elo rating system predates this corpus entirely: it was devised by Arpad Elo for ranking chess players from pairwise game outcomes and has since been adopted broadly for ranking anything compared only in pairs, including language models. Each competitor $i$ has a rating $R_i$, and the probability that $i$ beats $j$ is modeled as a logistic function of the rating gap: $$P(i \text{ beats } j) = \frac{1}{1 + 10^{(R_j - R_i)/400}}.$$ After an outcome is observed, ratings move toward the gap between the observed score $S_i \in \{0, 0.5, 1\}$ and the expected score $E_i = P(i \text{ beats } j)$: $R_i \leftarrow R_i + K(S_i - E_i)$ for some step size $K$. Only differences between Elo scores carry meaning; the scale has no intrinsic zero, so one snapshot's rating has to be pinned by convention — Constitutional AI pins its SL-CAI initial snapshot to zero.
How Constitutional AI computes it
Constitutional AI computes separate helpfulness and harmlessness Elo scores from crowdworker pairwise model-comparison preferences, following the same procedure as Bai et al. (2022), to track 24 model snapshots across training (10,274 helpfulness and 8,135 harmlessness comparisons collected in total) (constitutional-ai, §"3.3 Main Results", p. 8). Plotting harmlessness Elo against RL training sequence produces the paper's Figure 3, and plotting harmlessness against helpfulness for all snapshots at once produces Figure 2, the plot on which Pareto Improvement (helpfulness vs. harmlessness) is read off directly (constitutional-ai, §"4.3 Main Results", p. 12).
A generalized, and not fully neutral, instrument
Where InstructGPT's Win rate (preference against baseline) compares each model against one fixed anchor, Constitutional AI needs to compare many training trajectories at once, so the Elo score generalizes win rate onto a single shared interval scale (Elo Score (helpfulness/harmlessness) — generalizes → Win rate (preference against baseline)) — at the cost of that arbitrary zero point and a scale that moves with elicitation instructions. Concretely, crowdworkers computing the harmlessness Elo were newly instructed to prefer less evasive responses whenever two replies were equally harmless, which the paper credits with shrinking the harmlessness gap between the Helpful and HH RLHF baselines relative to Bai et al.'s (2022) earlier numbers under the old instructions (Elo Score (helpfulness/harmlessness) — was tuned to penalize → Evasiveness, constitutional-ai, §"1.1 Motivations", p. 3). Because Elo alone cannot say whether any model is safe in absolute terms, Absolute Harmfulness Score is trained as an independently-anchored check, and the two instruments are shown agreeing on the paper's central claim in Figure 10 (Absolute Harmfulness Score — cross checks → Elo Score (helpfulness/harmlessness)). The instrument therefore has two lives in this corpus: an explanatory analogy in 2017, borrowed to justify a loss function, and a literal, computed, and imperfectly neutral ranking metric in 2022. These threads sit inside the Automatic overlap scores versus human and preference judgments and The helpfulness-harmlessness tradeoff and CAI's claim to beat it themes, and the A primary metric, and a second instrument built to catch what it can't, One paper's central result, traced edge by edge, and Connections the source papers never draw, visible only by reading them together connective threads.