Adversarial examples
Concrete Problems in AI Safety — inherited
Inputs, often imperceptibly perturbed, that cause a learned model such as a neural network to make confidently incorrect predictions.
A photograph altered by amounts no human could notice can make a classifier confidently wrong. Such inputs, adversarial examples, were documented in deep networks around 2013 and belong to the wider machine learning literature rather than to this corpus.
The vulnerability itself is the opening move, followed by the reason a paper about reward hacking needed it: defenses built for classifiers become a template for defending reward functions too. Counterexample resistance, the remedy that makes the transplant explicit, is worth following next.
A vulnerability that predates this corpus
Adversarial examples are inputs to a learned model, most often a classifier, that have been perturbed, frequently by an amount imperceptible to a human, so as to cause a confident, incorrect prediction. The phenomenon was documented in deep image classifiers starting around 2013-2014, when researchers showed that small, carefully chosen perturbations could reliably flip a network's output while leaving the image visually unchanged. Formally, given a classifier $f_\theta$ correctly labeling input $x$ as $y$, an adversarial example is a perturbation $x' = x + \delta$ with $\|\delta\| \le \epsilon$ for some small $\epsilon$, such that $f_\theta(x') \neq y$. One standard construction, the fast gradient sign method, moves $x$ a fixed step in the direction that most increases the model's loss, $$\delta = \epsilon \cdot \mathrm{sign}\!\left(\nabla_x \, \mathcal{L}(\theta, x, y)\right),$$ exploiting the fact that even a highly accurate model's decision boundary can sit close to almost every real input once the input space is high-dimensional. Concrete Problems in AI Safety inherits this concept rather than introducing it, citing it as background (concrete-problems, §"Abstract Rewards:", p. 8; §"Counterexample Resistance:", p. 10).
Why a reward-hacking paper needed it
The paper's reward-hacking taxonomy includes Abstract rewards (reward hacking cause), the observation that sophisticated reward functions will need learned, high-dimensional components, precisely the kind of model susceptible to this vulnerability. That link motivates Counterexample resistance, the proposed remedy of applying existing adversarial-robustness defenses, adversarial training and weight uncertainty, to a reward model's learned components (see Adversarial examples — motivates but only partially covers → Counterexample resistance). The paper is explicit that the transplant is partial: "adversarial counterexamples are just one manifestation of reward hacking, so counterexample resistance can only address a subset of these potential problems" (concrete-problems, §"Counterexample Resistance:", p. 10). Adversarial examples are a narrow, well-defined vulnerability, a classifier's output changing sharply under a small targeted perturbation, while reward hacking is broader, encompassing failures like feedback loops or environmental tampering that involve no perturbed input and no misclassification at all.