Generative adversarial networks (GANs)
Concrete Problems in AI Safety — inherited
A prior neural-network training framework (Goodfellow et al. 2014) that pits a generator against a discriminator, cited as the inspiration for adversarial reward functions.
Generative adversarial networks, introduced by Goodfellow and colleagues in 2014, train a generator by pitting it against a discriminator that tries to tell its outputs from real data -- learning framed as a two-player game.
The minimax mathematics anchors the page, and then comes the question of what transfers to reward hacking and what does not. Adversarial reward functions, the proposal this game directly inspired, is the natural next stop.
A neural-network training framework that predates the paper
Generative adversarial networks were introduced by Goodfellow et al. (2014), several years before Concrete Problems, as a framework for training a generative model by pitting it against an adversary; the paper cites this prior work directly as the inspiration for one of its own reward-hacking remedies (concrete-problems, §"Adversarial Reward Functions:", p. 9).
The mathematics: a minimax game between generator and discriminator
A GAN consists of a generator $G$, which maps noise $z \sim p_z$ to samples $G(z)$ intended to resemble real data, and a discriminator $D$, which outputs the probability that a given sample came from the real data distribution $p_{\text{data}}$ rather than from $G$. The two are trained jointly on the minimax objective $$\min_G \max_D \; \mathbb{E}_{x \sim p_{\text{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))],$$ with $D$ trained to maximize its ability to distinguish real from generated samples and $G$ trained to minimize that same quantity, i.e. to fool $D$. At the game's theoretical equilibrium, $G$'s output distribution matches $p_{\text{data}}$ exactly and $D$ can do no better than chance, $D(x) = 1/2$ everywhere. What makes the framework distinctive is that neither network is a fixed target: both improve together, so the pressure on $G$ to generate convincing samples grows precisely as fast as $D$'s ability to catch it.
What transfers to reward hacking, and what does not
Concrete Problems' Adversarial reward functions proposal is explicitly modeled on this shape: instead of a static reward function sitting still to be gamed, a second agent actively searches for scenarios the primary system rates highly but a human would rate poorly, the same generator-versus-discriminator structure (concrete-problems, §"Adversarial Reward Functions:", p. 9). But the analogy does not transfer cleanly. In a GAN, generator and discriminator are trained symmetrically from the same gradient signal and can be arbitrarily matched in capability, part of why the game is stable. The reward-checking agent in adversarial reward functions cannot be symmetric in the same way: the paper notes it "would have to ensure that the reward-checking agent is more powerful, in a somewhat subtle sense, than the agent that is trying to achieve rewards" (concrete-problems, §"Adversarial Reward Functions:", p. 9), an asymmetric power requirement with no GAN analogue and no proposed mechanism for guaranteeing it holds as the reward-seeking agent scales up. Within the theme Borrowing adversarial ML to defend against reward hacking, generative adversarial networks function as the borrowed conceptual template, alongside Adversarial examples, for a family of proposed defenses, Adversarial reward functions, Adversarial blinding, and Counterexample resistance, that treat reward hacking as structurally akin to adversarial robustness.