Compute vs. Human Feedback Cost Analysis
Deep Reinforcement Learning from Human Preferences — introduced
A cost analysis (footnote) estimating that the Atari experiments' compute cost (about $25 for roughly a day of training on a 16-CPU, one-GPU cloud VM) is already comparable to the cost of the non-expert human feedback used (about $36 at US minimum wage for 5,000 labels), used to argue that further reductions in the number of human labels required would yield diminishing practical returns.
It's tempting to assume that cutting the number of human labels needed is always worth chasing further. A footnote tucked into the paper's discussion section quietly complicates that assumption with one back-of-envelope calculation.
The page walks through the footnote's own numbers, cloud compute against minimum-wage labor for one Atari run, then the argument built on them, that once the two costs sit close together, squeezing labels further saves less than it seems, and closes on where the estimate does and doesn't generalize.
What the footnote computes
Tucked into the discussion section as a footnote, this is a back-of-envelope cost comparison rather than a formal result: "For the Atari experiments we are using a virtual machine with 16 CPUs and one Nvidia K80 GPU which costs ~$700/month on GCE. Training takes about a day, so the compute cost is ~$25. Training with 5k labels corresponds roughly to 5 hours of human labour, at US minimum wage this totals ~$36" (deep-rl-human-prefs, §"4 Discussion and Conclusions", p. 11). The two numbers, roughly $25 of cloud compute and roughly $36 of non-expert human labor for a single Atari run, land close enough together to support the paper's point without needing precision.
The argument it is used to make
The estimate answers an implicit objection to the paper's framing: if the point of learning from comparisons is to save expensive human attention, why not push the label count down further? The Human-Feedback Sample Efficiency edge shows how directly this footnote answers that question: right after quantifying the roughly three-orders-of-magnitude reduction in interaction complexity, the paper states it is "already hitting diminishing returns on further sample-complexity improvements" (deep-rl-human-prefs, §"4 Discussion and Conclusions", p. 10) precisely "because the cost of compute is already comparable to the cost of non-expert feedback" (deep-rl-human-prefs, §"4 Discussion and Conclusions", p. 11). Once compute and labor cost sit at roughly the same order of magnitude, shaving labels further would save money on one side of a ledger that is already balanced, rather than unlocking a fundamentally cheaper method.
Its place in the paper's economics
This is the paper's only explicit dollar figure, and it sits alongside Contractor Preference-Labeling Protocol and Clip-Length Effects on Human Evaluation as one of the ways the paper reasons about how expensive a human rater actually is per unit of information extracted. Unlike the sample-efficiency headline, this analysis is scoped specifically to the Atari setup (a 16-CPU, one-GPU cloud VM and 5,000 labels), and the paper does not claim the same cost comparison would hold for the MuJoCo robotics tasks, which used far smaller label budgets of 350-1400 queries.