Atari games (as an RL testbed)
Concrete Problems in AI Safety — inherited; Deep Reinforcement Learning from Human Preferences — inherited
A widely used prior benchmark suite of video games used in the paper as a toy environment for proposed semi-supervised RL and reward hacking experiments.
Atari's old video games found a second life as a standard proving ground for reinforcement learning. On this wiki they appear not as a technique but as a testbed: Concrete Problems in AI Safety (2016) borrows them as a toy world for its oversight and reward-hacking experiments.
First the page explains how a pre-existing benchmark was repurposed as a difficulty ladder, then what that enables: a graded test of how little reward feedback an agent can manage on.
A pre-existing benchmark repurposed as a difficulty ladder
Atari 2600 games were not created for this paper; they were already a widely used reinforcement-learning benchmark by 2016, popularized as a standard testbed once deep Q-networks were shown to learn to play many of the games directly from pixels. Concrete Problems inherits this benchmark and puts it to a new use: a toy domain for testing proposed solutions to scalable oversight, the problem of ensuring safe behavior when the true objective is too costly to evaluate on every training example (concrete-problems, §"5 Scalable Oversight", pp. 12-13).
What it enables: a graded test of oversight scarcity
The paper does not propose one Atari experiment but a sequence of them, each stripping away a bit more of the oversight the agent would ordinarily have. The base case has a semi-supervised RL agent learn to play "relying almost entirely on the visual display of the score" from only a small number of direct reward signals; harder variants let the agent modify the displayed score without touching the real score, require it to pause the game just to see its score, force it to infer reward from indirect cues like sounds, or reduce it to answering explicit queries such as "how many points did I get when that enemy ship blew up?" (concrete-problems, §"5 Scalable Oversight", pp. 12-13). Atari's appeal here is structural: because the same environment supports all of these variants, the benchmark isolates which specific failure mode, score tampering, sparse observation, indirect cues, or query-based inference, a proposed semi-supervised RL method actually survives, turning one familiar testbed into a graded difficulty ladder rather than a single pass or fail test. This is why the concept is the proposed testbed for the theme Scaling oversight by predicting reward on unlabeled episodes, grouped with Supervised reward learning, Semi-supervised or active reward learning, Unsupervised value iteration, and Unsupervised model learning, the four concrete algorithmic proposals the Atari ladder is meant to exercise, all under the umbrella of Scalable oversight.
The 2017 paper's second testbed
The benchmark gets its first concrete deep-RL run the following year. Atari is one of the 2017 paper's two experimental domains, alongside MuJoCo, drawing seven games, BeamRider, Breakout, Enduro, Pong, Qbert, SpaceInvaders, and Seaquest, from the set Mnih et al. (2013) studied, accessed through the Arcade Learning Environment via OpenAI Gym (deep-rl-human-prefs, §"3.1.2 Atari", p. 7). Because the games display a running score onscreen, which could otherwise let the very reward the experiment is meant to withhold be trivially inferred, the paper replaces the score area with a constant black background on all seven games, additionally blanking BeamRider's enemy-ship count and Enduro's speedometer (deep-rl-human-prefs, §"A.2 Atari", p. 15). This is the same environment active reward learning's ensemble-disagreement query selection is exercised in, and it is also where the paper's Reward hacking (avoiding) sighting occurs, on Pong specifically, under the "no online queries" ablation (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 9).