Counterexample resistance
Concrete Problems in AI Safety — introduced
A proposed remedy applying existing defenses against adversarial examples, such as adversarial training and weight uncertainty, to harden learned reward components against exploitation.
Image classifiers can be hardened against adversarial examples, so the learned parts of a reward function ought to be hardenable the same way -- adversarial training, weight uncertainty, and their kin. That transplant is counterexample resistance.
The transplant gets the fullest treatment, separated along the way from its adversarial neighbors, before the page explains why the corpus treats it as a hub rather than a single fix. Adversarial blinding and adversarial reward functions are both worth a look, each borrowing the same lineage differently.
Transplanting adversarial defenses into reward models
Counterexample resistance proposes to harden learned components of a reward function the same way adversarial training and weight uncertainty harden an image classifier against Adversarial examples (concrete-problems, §"Counterexample Resistance:", p. 10). It is Concrete Problems' most explicit transplant from adversarial machine learning into the Reward hacking (avoiding) problem, and the paper is careful about scope: it targets specifically the Abstract rewards (reward hacking cause) cause, where a reward function is itself a learned, high-dimensional model that can literally have adversarial examples in the classifier sense. The paper states the limit directly: "adversarial counterexamples are just one manifestation of reward hacking, so counterexample resistance can only address a subset of these potential problems" (concrete-problems, §"Counterexample Resistance:", p. 10), a caveat documented in Adversarial examples — motivates but only partially covers → Counterexample resistance.
Distinguishing it from its two neighbors
Counterexample resistance sits in the paper's list of remedies next to two other concepts labeled "adversarial," and the theme Borrowing adversarial ML to defend against reward hacking groups all three, but the relationships are not interchangeability. Against Adversarial blinding, Adversarial blinding — borrows from a different adversarial ml lineage than → Counterexample resistance shows the two descend from different corners of adversarial ML: counterexample resistance from robustness training that defends a prediction, adversarial blinding from invariance training that removes information altogether. Against Adversarial reward functions, Adversarial reward functions — operates at a different layer than → Counterexample resistance shows the two intervene at different layers, the reward function's outer search loop versus its inner learned components, and are complementary: an outer-loop reward-search agent gains little if its own inner classifier can still be fooled by a crafted counterexample.
Why the corpus treats it as a hub, not a single fix
Counterexample resistance is the piece that would have to work for the other two adversarial remedies' guarantees to mean anything, which is what makes it the busiest node in Borrowing adversarial ML to defend against reward hacking: three separate edges connect it to Adversarial examples, Adversarial blinding, and Adversarial reward functions, more than any other member of that theme. It also appears in both connective themes built on top of that theme, Borrowed machinery that doesn't fully transfer, and says so, which tracks where a borrowed technique's fit is only partial, and A footnote or related-work list implies two things are equivalent; they aren't, which tracks where a shared label invites treating structurally different remedies as one idea. Read together, those two connective themes make the same point from opposite directions: counterexample resistance looks, on the page, like the most literal adversarial-ML transplant in the paper, and it is also the remedy whose scope the paper is most explicit about bounding.