Adversarial reward functions
Concrete Problems in AI Safety — introduced
A proposed remedy making the reward function itself an active agent that searches for scenarios the ML system rates highly but a human rates poorly, rather than a static target to be gamed.
A static reward function cannot fight back while an agent probes it for loopholes. The adversarial-reward-functions proposal makes the reward function an agent in its own right -- one that actively hunts for cases the system rates highly but a person would not.
The fighting-back idea is the opening move, followed by a distinction that matters in practice: the adversary runs in the outer training loop, not the inner one. Generative adversarial networks, its direct inspiration, connects out from here.
Making the reward function fight back
Adversarial reward functions is Concrete Problems' proposal to stop treating a reward function as "a static object that has no way of responding" to an agent's attempts to game it (concrete-problems, §"Adversarial Reward Functions:", p. 9). Instead, a second agent actively searches for scenarios the primary system rates highly but a human would rate poorly, the generator-versus-discriminator shape borrowed explicitly from Generative adversarial networks (GANs). The paper flags the one place the analogy strains: a GAN's two networks are trained symmetrically and can be arbitrarily matched in capability, whereas an adversarial reward function needs the reward-checking agent to be "more powerful, in a somewhat subtle sense, than the agent that is trying to achieve rewards" (concrete-problems, §"Adversarial Reward Functions:", p. 9), an asymmetric requirement with no GAN analogue and no proposed mechanism for guaranteeing it as the reward-seeking agent scales (see Generative adversarial networks (GANs) — inspires but does not transfer cleanly to → Adversarial reward functions).
Outer loop, not inner loop
Among the paper's Reward hacking (avoiding) remedies, adversarial reward functions and Counterexample resistance both borrow adversarial-ML machinery but intervene at different layers, a relationship made explicit in Adversarial reward functions — operates at a different layer than → Counterexample resistance. Adversarial reward functions attacks the outer loop, turning the whole reward signal into a search agent so it stops being a fixed target the policy can map out at leisure. Counterexample resistance attacks the inner loop, hardening whichever learned components sit inside the reward computation itself. The two are complementary rather than substitutes: a reward-search agent built from a classifier still vulnerable to adversarial counterexamples gains little from being adversarial in the outer sense alone.