TruthfulQA
Training language models to follow instructions with human feedback — inherited; Steering Llama 2 via Contrastive Activation Addition — inherited
A benchmark of short-answer questions (Lin et al., 2021) designed to test whether a model generates truthful and informative answers rather than mimicking common human falsehoods.
Some questions invite wrong answers because people keep repeating them wrong: urban myths, garbled science, comfortable falsehoods. TruthfulQA tests whether a language model echoes those popular mistakes or manages to answer truthfully.
After introducing the benchmark, the page turns to the two prompting regimes InstructGPT is evaluated under, a distinction that matters more than it sounds, then to a 2023 use that gives the benchmark a job it was not designed for: checking whether steering a model's sycophancy also shifts how often it repeats a popular falsehood.
The benchmark
TruthfulQA (Lin et al., 2021) is a benchmark of 817 short-answer questions constructed to probe whether a language model reproduces common human misconceptions and falsehoods rather than answering truthfully; its questions are adversarially selected to be ones on which models tend to mimic popular false beliefs (e.g. urban myths or misunderstood science) rather than state the correct answer. TruthfulQA predates InstructGPT and is inherited here unmodified as the paper's primary open-domain truthfulness probe (instructgpt, §"4.2 Results on public NLP datasets", p. 13; §"TruthfulQA", p. 46).
Two prompting regimes
InstructGPT evaluates under two prompt formats: a plain few-shot QA prompt, and a helpful "Instruction+QA" prompt, following Lin et al. (2021), that explicitly permits the model to answer "I have no comment" when it is not certain, rather than confidently asserting a falsehood (instructgpt, §"TruthfulQA", p. 46). Under this instructed setting, PPO models "err on the side of being truthful and uninformative rather than confidently saying a falsehood," a discipline the baseline GPT-3 model exhibits less well (instructgpt, §"4.2 Results on public NLP datasets", p. 13). The paper reports its InstructGPT models generate truthful and informative answers roughly twice as often as GPT-3 on this benchmark, with results holding even restricted to the subset of questions not adversarially selected against GPT-3 (instructgpt, §"1 Introduction", p. 3). Together with the Hallucination (closed-domain fabrication) metric on closed-domain tasks, TruthfulQA supplies this paper's other major piece of evidence for its truthfulness claims, and is a member of the Operationalizing truthfulness and social bias into measurable benchmarks theme built to catch harms that general capability benchmarks would miss.
2023: reused as a diagnostic for the sycophancy hypothesis
Steering Llama 2 via Contrastive Activation Addition (2023) puts TruthfulQA to a different use: not as a standalone benchmark score but as an instrument for testing a causal hypothesis about a different behavior. Appendix H reasons that if sycophancy is the model "misgeneralizing its RLHF training objective as 'sounding good to the user' instead of truthfully reflecting its internal world model," then steering with the Sycophancy vector should move TruthfulQA performance in a specific direction, and the paper reformats TruthfulQA's questions into the same two-option A/B format used throughout its other evaluations to make that comparison possible (contrastive-activation-addition, §"H Sycophancy steering and TruthfulQA", p. 14). Broken down by category, the effect is small but lands where the hypothesis predicts: in Llama 2 13B Chat, subtracting the sycophancy vector improves average TruthfulQA performance by 0.02 and adding it worsens performance by 0.03; in the 7B model, subtracting improves performance by 0.01 and adding worsens it by 0.05 (contrastive-activation-addition, §"H Sycophancy steering and TruthfulQA", p. 14). The paper is candid that the effect size is small enough to need further investigation, so this use of TruthfulQA reads as suggestive rather than confirmatory — a benchmark built in 2021 to score truthfulness once, repurposed here as a probe for whether a specific mechanistic story about a specific failure mode holds up, without displacing its primary identity in this corpus as InstructGPT's open-domain truthfulness benchmark.