T0 (instruction-tuned baseline)
Training language models to follow instructions with human feedback — inherited
A compilation of NLP tasks with natural-language instructions (Sanh et al., 2021), specifically the T0++ variant, fine-tuned into a 175B GPT-3 baseline for comparison against InstructGPT.
T0 belongs to the same family as FLAN: take a large compilation of NLP tasks, dress each in instruction templates, and fine-tune until the model generalizes to unseen tasks. InstructGPT rebuilds the recipe, the larger T0++ variant, on a 175-billion-parameter GPT-3.
InstructGPT rebuilds this same recipe at scale to serve as the compilation-based rival its directly trained models are measured against, and beat. FLAN, its closest relative, makes the natural next stop.
The baseline, in general
T0 (Sanh et al., 2021) is a T5-based model fine-tuned on a large compilation of NLP tasks, each reformatted with natural-language instruction templates, designed to test whether multitask instruction tuning enables zero-shot generalization to unseen tasks. The larger T0++ variant, used in this paper, comprises roughly 96 million training datapoints. T0 predates InstructGPT and is inherited here purely as a comparison baseline rather than a technique the paper builds on.
Its role: a compilation-based rival InstructGPT beats
InstructGPT's authors fine-tune a 175B GPT-3 model on T0++ data, subsampled from 96 million down to about 1 million datapoints, as detailed by the T0 (instruction-tuned baseline) — is scaled to match → FLAN (instruction-tuned baseline) edge, specifically to keep this baseline comparable in scale to the similarly constructed FLAN (instruction-tuned baseline) baseline rather than letting either win purely on data volume (instructgpt, §"C.5 FLAN and T0 models", p. 42). Both T0 and FLAN checkpoints were selected by the same rule used for the Supervised fine-tuned (SFT) model and the GPT-3-prompted prefix, highest reward-model score on a validation set rather than lowest validation loss, tying all of the paper's comparison baselines to one consistent selection criterion. On the paper's own evaluation, InstructGPT is preferred over this T0 baseline 79 ± 4% of the time (instructgpt, §"1 Introduction", p. 4; §"C.5 FLAN and T0 models", p. 42), a result the paper attributes to public NLP compilations under-representing the open-ended generation and brainstorming tasks that dominate real API usage (instructgpt, §"4.1 Results on the API distribution", p. 13). T0 is a member of the Instruction following as an alternative to next-token prediction theme, which frames both fixed-compilation baselines like T0 and FLAN, and prompting-only baselines like GPT-3-prompted, as instructive rivals against which InstructGPT's preference-learning approach to instruction-following is argued for.