Operationalizing truthfulness and social bias into measurable benchmarks
A model can ace every capability benchmark and still repeat popular falsehoods, turn toxic, or lean on a stereotype, none of which shows up in an overlap score. This theme covers the purpose-built instruments this corpus uses to catch exactly those harms, along with the scoring machinery, an outside toxicity API and a purpose-built entropy metric, needed to make their numbers mean anything.
Winogender and CrowS-Pairs pair off neatly, both leaning on the same entropy-based bias metric to turn a preference between two sentences into a single score, while truthfulness and toxicity get their own dedicated tests. A later paper in the corpus reaches for one further purpose-built instrument, a hallucination benchmark, to check that its own findings hold up against a standard question set nobody on the team wrote.
Alongside general capability, InstructGPT is evaluated on a distinct set of benchmarks purpose-built to catch harms that overlap-based capability metrics would miss. TruthfulQA tests whether a model gives truthful, informative answers rather than mimicking common human falsehoods. RealToxicityPrompts supplies sentence-starts with varying toxicity to test whether a model continues text toxically, scored automatically via the third-party Perspective API and cross-checked against human toxicity ratings. Winogender and CrowS-Pairs both probe social bias through paired sentences, one via coreference-style completions, the other via stereotype-versus-anti-stereotype pairs, and both are scored with the paper's own entropy-based bias metric, which converts a model's relative preference between the two sentences in a pair into a binary distribution and reports its entropy, so a lower score indicates a stronger, more biased preference. This theme's claim is that truthfulness and bias needed dedicated instruments; the general NLP benchmark suite could not have caught either. HaluEval extends that instrument-building lineage to hallucination specifically: a 2023 hallucination-detection benchmark whose QA split's first 1,000 questions the 2025 persona-vectors paper borrows as an external, out-of-distribution question set, confirming that the hallucination trait expression scores it measures on its own evaluation questions generalize to a standard, independently constructed benchmark rather than being an artifact of the paper's own question design.