Adversarial blinding
Concrete Problems in AI Safety — introduced
A proposed remedy using adversarial training techniques to prevent an agent from understanding or gaining mutual information with part of its environment, such as how its reward is generated.
Some exploits require understanding; an agent cannot game a reward process it genuinely cannot model. Adversarial blinding proposes using adversarial training to strip an agent of information about selected parts of its environment -- such as how its reward is generated.
The blinding idea gets explained first, distinguished from a close neighbor, counterexample resistance, which hardens rather than hides. That neighbor is worth following, for what the two approaches share and where they part.
Blinding an agent to what it could exploit
Adversarial blinding is a remedy Concrete Problems proposes for Reward hacking (avoiding), applying adversarial-training techniques not to harden a prediction but to strip a model of mutual information with some variable altogether (concrete-problems, §"Adversarial Blinding:", p. 10). Applied to reward hacking, the goal is to make it "impossible for an agent to understand some part of its environment, or even to have mutual information with it," most importantly how its own reward is generated, so an agent cannot exploit a mechanism it is structurally prevented from perceiving. The paper's own description of the technique, "cross-validation for agents," signals that its intended function is diagnostic isolation rather than robustness in the usual adversarial-ML sense.
A different adversarial lineage than its neighbor
Adversarial blinding sits in the paper's list of remedies next to Counterexample resistance, and both are labeled "adversarial," which invites treating them as variations on one idea. Adversarial blinding — borrows from a different adversarial ml lineage than → Counterexample resistance shows they are not: counterexample resistance descends from the Adversarial examples literature and hardens a model's output against worst-case input perturbations, while adversarial blinding descends from domain-adversarial and invariance training, where the adversarial objective removes information rather than defending a prediction. One hardens a model against being fooled; the other prevents it from perceiving the thing it might exploit. The theme Borrowing adversarial ML to defend against reward hacking groups both alongside Adversarial reward functions as the corpus's transplant of adversarial ML into the reward-hacking problem.