A small model wins, and the paper shows exactly which rung the win comes from
A hundred-times-smaller InstructGPT still comes out ahead of GPT-3, and the paper is explicit about why: the same scale asymmetry repeats at every stage beneath that headline number. A reward model nearly thirty times smaller than the policy it supervises somehow steers it; ten written principles substitute for tens of thousands of comparisons.
A ladder of baselines is built for exactly this claim: T0 scaled to match FLAN so neither wins on data volume, FLAN setting the comparison floor, and a prompted GPT-3 showing how far prompting alone falls short, each rung isolating one specific reason the last one failed. The same ladder logic turns up twice more in this corpus, once in 2017 when a policy-gradient method closes an exploration gap against a value-based one, and once in 2023 when a steering paper settles which prompting method should serve as its own baseline before the real comparison begins.
InstructGPT outperforms GPT-3 at 100x smaller scale, and the paper is explicit that this rests on a scale asymmetry repeated throughout the pipeline: the reward model supervises PPO training from below, a judge nearly 30x smaller than the policy it steers, and the constitution later replaces human harm labels in Constitutional AI with roughly ten principles standing in for tens of thousands of comparisons, while the API prompt distribution supplies label-free scale to the cheapest stage of the pipeline. The paper stages its headline claim as a ladder rather than a single number: T0 is scaled to match FLAN so neither baseline wins on data volume alone; FLAN sets the comparison floor for instruction following because its fixed task compilation underrepresents real usage; and the GPT-3-prompted baseline shows the limits of prompting alone, trailing trained instruction-following substantially even when optimized against the very reward model used for RLHF itself. Each rung isolates a different specific reason the previous one falls short. The 2017 paper stages an analogous ladder rung five years earlier, an algorithm-class asymmetry rather than a parameter-count one: A2C closes its Enduro exploration gap with DQN, matching a value-based method's previously reported score not by outperforming it on the merits of policy-gradient learning but because human-shaped preference feedback supplies exactly the dense progress signal random policy-gradient exploration could never find on its own. As with T0-versus-FLAN or the GPT-3-prompted baseline, the comparison isolates one specific bottleneck, here exploration under sparse reward, rather than asserting a general win, and DQN itself was never retrained for the comparison; its number is simply cited from prior work, the same citation-not-rerun structure the paper's other baseline rungs share. CAA adds a rung of its own, six years later and over a technique choice rather than a model or algorithm: few-shot prompting loses out to system-prompting when the paper tests which prompting method should serve as CAA's baseline, a single instructional system prompt beating few-shot examples at steering any of the seven behaviors tested. As with T0-versus-FLAN, the comparison isolates one specific variable, here which prompting technique to standardize on, rather than asserting a general verdict on few-shot prompting; the paper is explicit that it never exhaustively searched either configuration, so the gap it reports is a property of this comparison, not a settled fact about the technique. Every later baseline number the paper reports simply as "prompting" means this winning configuration, system-prompting, because that is the rung the ladder settled on before the paper's real comparisons, against activation steering, begin.