BLEU
Training language models to follow instructions with human feedback — inherited
A standard machine-translation quality metric used to evaluate model outputs on the WMT 2015 French-to-English translation task.
BLEU has graded machine translation since 2002 on a blunt principle: count how many short word sequences a candidate translation shares with human references. It rewards overlap, not understanding, and remains the standard score for tasks like the WMT benchmark used here.
Its role here is narrow, scoring one translation task, but the absence of any concept connections is itself telling: BLEU answers to the field at large, not to this paper's particular argument.
What the metric measures
BLEU (Bilingual Evaluation Understudy) is a precision-based automatic metric for machine translation quality, comparing a candidate translation against one or more human reference translations by n-gram overlap. For n-gram order $n$, the modified precision $p_n$ counts how many n-grams in the candidate also appear in a reference (each reference n-gram usable only as many times as it occurs in the reference, to prevent repeating a single correct word to inflate the score), divided by the total number of candidate n-grams. Because precision alone rewards short, safe outputs, BLEU multiplies the geometric mean of these precisions by a brevity penalty that punishes candidates shorter than the reference: $$\text{BLEU} = BP \cdot \exp\left(\sum_{n=1}^{N} w_n \log p_n\right), \qquad BP = \begin{cases} 1 & \text{if } c > r \\ e^{1-r/c} & \text{if } c \le r \end{cases}$$ where $c$ is the candidate length, $r$ the reference length, and $w_n$ typically uniform weights over $n=1,\ldots,4$. BLEU predates this paper by roughly fifteen years and is one of the oldest standard metrics in machine translation evaluation; InstructGPT inherits it unmodified.
Its role in this corpus
BLEU is used to score model outputs on the WMT 2015 French-to-English translation task, one of the fixed public benchmarks in the automatic evaluation suite (instructgpt, §"WMT Fr → En 15", p. 49). Within the Automatic overlap scores versus human and preference judgments theme it stands alongside ROUGE-L and F1 score (span-overlap QA metric) as a reference-overlap metric, cheap to compute but blind to whether an output is actually preferred, in contrast to the judgment-based Win rate (preference against baseline) and Likert overall-quality rating the paper uses for its headline claims. Translation is also one of the tasks where the Alignment tax proves hardest to pay down: the paper reports that "the performance of the PPO-ptx model still lags behind GPT-3 on DROP, SQuADv2, and translation" even after pretraining-gradient mixing is applied (instructgpt, §"4.2 Results on public NLP datasets", p. 15), making BLEU on this benchmark one of the corpus's concrete pieces of evidence that RLHF's capability regressions are not uniformly recoverable.