BIG-Bench
Constitutional AI: Harmlessness from AI Feedback — inherited
A large collaborative benchmark suite for language models that hosts the original HHH evaluation set used as a comparison baseline.
A benchmark built by one lab tends to reflect that lab's blind spots. BIG-Bench is the opposite: a sprawling, many-author collection of tasks meant to probe language models more broadly than any single benchmark could, and it appears here mainly because of one thing it happens to host.
Its own breadth is introduced first, before narrowing to the one part relevant here, the original HHH evaluation questions, and connecting out to that evaluation set.
A large, crowd-built benchmark suite, predating this paper
BIG-Bench ("Beyond the Imitation Game Benchmark," Srivastava et al. 2022) is a large, collaboratively assembled benchmark suite for language models, contributed to by researchers across many institutions and comprising a broad collection of diverse tasks intended to probe capabilities and limitations that narrower, single-author benchmarks tend to miss. It predates Constitutional AI and is not introduced by any of the three papers in this corpus; Constitutional AI uses it only as a host for one specific evaluation set.
Hosting the original HHH evaluation
BIG-Bench's relevance to this corpus is narrow but concrete: it is where the original 221-comparison HHH Evaluation Set (binary comparisons), first written by Askell et al. (2021) to test the Helpful, honest, harmless (HHH) alignment framework, was made publicly available (HHH Evaluation Set (binary comparisons) — is hosted as a subset of → BIG-Bench; constitutional-ai, §"B Identifying and Classifying Harmful Conversations", p. 20). Constitutional AI uses this BIG-Bench-hosted set as a comparison baseline, reporting that models "can now achieve well over 90% binary accuracy" on it, which is precisely why the paper goes on to write 217 additional, harder comparisons of its own rather than continuing to rely on the saturated original set. Those 217 new comparisons are not added back into BIG-Bench; they exist only in the paper's own repository, so "the HHH eval" as used by Constitutional AI is a composite of a small, versioned public subset hosted inside BIG-Bench and a private extension attached to this one paper, a distinction relevant to anyone trying to reproduce the evaluation later.