Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gradient hacking: definitions and examples, published by Richard Ngo on June 29, 2022 on The AI Alignment Forum. Gradient hacking is a hypothesized phenomenon where: A model has knowledge about possible training trajectories which isn’t being used by its training algorithms when choosing updates (such as knowledge about non-local features of its loss landscape which aren’t taken into account by local optimization algorithms). The model uses that knowledge to influence its medium-term training trajectory, even if the effects wash out in the long term. Below I give some potential examples of gradient hacking, divided into those which exploit RL credit assignment and those which exploit gradient descent itself. My concern is that models might use techniques like these either to influence which goals they develop, or to fool our interpretability techniques. Even if those effects don’t last in the long term, they might last until the model is smart enough to misbehave in other ways (e.g. specification gaming, or reward tampering), or until it’s deployed in the real world—especially in the RL examples, since convergence to a global optimum seems unrealistic (and ill-defined) for RL policies trained on real-world data. However, since gradient hacking isn’t very well-understood right now, both the definition above and the examples below should only be considered preliminary. RL credit hacking examples A policy might know that a certain sequence of actions leads to higher reward. But if it chooses not to take those actions, it won’t be rewarded for them, and that type of behavior won’t be reinforced (until it finds them via random exploration, which could take a very long time). A policy might know that humans give mistaken feedback on a given trajectory, and then select that trajectory more frequently in order to update itself in a certain direction. When humans give mistakenly positive feedback, this is just an example of specification gaming. But we could also imagine the model deliberately inducing mistakenly negative feedback. A policy might know which of its actions is actually responsible for receiving reward, even though the RL algorithm being used “doesn’t know”. Then the model can influence how much credit the RL algorithm gives different actions (e.g. by putting them earlier or later in the trajectory). This seems easier for dumb RL algorithms like REINFORCE, and seems harder for actor-critic methods. However, finding algorithms that make use of all the knowledge a model has about the training landscape seems like it’ll require a full solution to the ELK problem. A policy might know when it’s being trained off-policy versus on-policy (because off-policy it observes a trajectory of actions which it wouldn’t have taken), and choose actions differently in each case. A tentative possibility, related to the first example: during off-policy training, it could try novel “explore” actions which it knows would lead to high reward on-policy. Those actions would then be reinforced against (in favor of the action actually taken during that trajectory), since when performed off-policy the rewards are not actually seen. In multi-agent RL, even if rewards are zero-sum, different policies could coordinate to choose which of them will receive the reward in which episodes, to influence which behaviors are reinforced. Gradient descent hacking examples When local optima exist, a model might choose to do worse than it otherwise could, with the intention of changing its parameters in a certain direction. Intuitively speaking, gradient descent will then push in two directions: towards changing the parameters which implemented the decision to gradient hack, and towards changing the parameters which the model wanted to change (since their current values influenced its decision). Gradient d...