InstructGPT — outperforms at 100x smaller scale → GPT-3 (pretrained language model)

The comparison between these two models is a controlled experiment, not just a benchmark win: every InstructGPT model shares GPT-3's architecture and pretrained weights, so the only difference is fine-tuning on the human data pipeline. Under that control, labelers prefer 1.3B InstructGPT outputs to those of the 175B GPT-3, and 175B InstructGPT beats 175B GPT-3 head-to-head 85 ± 3% of the time (instructgpt, §"1 Introduction", p. 3). The paper also decomposes the gap into rungs — GPT-3, then GPT-3 with a crafted few-shot prefix, then SFT, then PPO — each adding a measurable step in labeler preference (instructgpt, §"4.1 Results on the API distribution", p. 11). The cost asymmetry is the deeper point: training the 175B PPO-ptx model took 60 petaflop/s-days against 3,640 for pretraining GPT-3 (instructgpt, §"5.1 Implications for alignment research", p. 17), so under 2% of pretraining compute bought more preference gain than a 100x parameter increase. On this distribution, alignment data, not scale, was the binding lever.