Reward autoencoder (goal transparency)
Concrete Problems in AI Safety — introduced
A proposed idea to interpret an agent's actions as an encoding of its reward function and apply autoencoding techniques so an external observer can decode what the agent is trying to do, implicitly penalizing side effects that obscure intent.
Watching what an agent does, could an observer decode what it is trying to do? The reward autoencoder treats an agent's actions as an encoding of its goal and asks that the encoding be easy to read -- legibility as a safety property.
The decoding idea opens the page, followed by a note that the penalty it creates is implicit rather than written down anywhere. It forms one half of a pair with CIRL, its natural next stop, each handling legibility from a different direction.
Making an agent's goal legible by decoding its actions
The reward autoencoder is a proposal introduced in Concrete Problems, credited in a footnote to Greg Wayne, for creating what the paper calls "goal transparency": treating an agent's actions as an implicit encoding of its reward function, then applying standard autoencoding techniques so that "an external observer can easily infer what the agent is trying to do" (concrete-problems, §"Multi-Agent Approaches:", p. 6). The idea borrows the encoder-decoder structure of a standard autoencoder, which learns to compress an input $x$ into a code $z = E(x)$ and reconstruct it, $\hat x = D(z)$, by minimizing reconstruction error $\|x - D(E(x))\|^2$; here the analogy treats the agent's observed behavior as the code and the agent's underlying goal as the thing that behavior should be decodable back into.
An implicit penalty rather than an explicit one
The mechanism is meant to create regularization without a hand-specified impact term: "actions that have lots of side effects might be more difficult to decode uniquely to their original goal, creating a kind of implicit regularization that penalizes side effects" (concrete-problems, §"Multi-Agent Approaches:", p. 6). An agent whose behavior is easy to reconstruct back into a single clean objective, in this framing, is one whose side effects have not obscured its intent; an agent whose actions are ambiguous between many possible goals, because it has disturbed the environment in ways unrelated to any one clean objective, is harder to decode and so implicitly penalized, without the designer ever having to specify what a side effect is.
One half of a two-directional pair
As a child of Multi-agent approaches (to side effects), the reward autoencoder is the paper's second concrete instantiation of making side-effect avoidance a matter of legibility between agent and human, alongside Cooperative Inverse Reinforcement Learning (CIRL), but it runs the legibility in the opposite direction. CIRL makes the human's goal legible to the agent, so the agent can act on inferred human intentions; the reward autoencoder makes the agent's goal legible to the human, so an outside observer can check what the agent is actually trying to do. Both belong to the theme Side effects and control as a relationship with other agents, and together they frame safety, for this part of the paper, as a two-way communication problem between agent and human rather than a one-sided optimization problem for the agent alone.