Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on gradient hacking, published by Richard Ngoon the AI Alignment Forum. Gradient hacking is the hypothesised phenomenon of a machine learning model, during training, deliberate thinking in ways which guide gradient descent to update its parameters in the directions it desires. The key intuition here is that because the loss landscape of a model is based on the cognition it does, models can make decisions for the purpose of affecting their loss landscapes, thereby affecting the directions in which they are updated. Evan writes: I don't think [gradient hacking] is actually that difficult once your deceptive model comes up with it as a strategy. First, note that since gradient descent is a local optimization process, it doesn't have to somehow change what the global minima might be, or even what the local minima are—it just has to create a ridge in the loss landscape that will push the gradient in the direction it wants. One example of this is for the model to check if its objective satisfies some criterion, and fail hard if it doesn't. Now, if the model gets to the point where it's actually just failing because of this, then gradient descent will probably just remove that check—but the trick is never to actually get there. By having such a check in the first place, the model makes it so that gradient descent won't actually change its objective, since any change to its objective (keeping all the other parameters fixed, which is what gradient descent does since it computes partial derivatives) would lead to such a failure. But I don't think this works as described, because of the way that gradient descent operates. Suppose that an agent has some trait which gradient descent is trying to push in direction x because the x-ness of that trait contributes to the agent’s high score; and that the agent wants to use gradient hacking to prevent this. Consider three possible strategies that the agent might try to implement, upon noticing that the x-component of the trait has increased: Deterministically failing hard above a sharp threshold of the x-component. In this case, the closer that gradient descent gets to the threshold, the better it will do. But since the threshold is sharp, the gradient will continue pointing towards it right until the agent hits the threshold. And since gradient update proceeds via discrete steps, then it will eventually cross over (causing the agent to fail) then bounce back, then cross over again, until something else changes. Deterministically increasing the extent to which it fails as the x-component increases. Here, once the cost of failure balances the benefit of increasing the x-component, I expect gradient descent to maintain roughly the same x-component, and the agent to continue failing in small ways, until something else changes. Gradually increasing its probability of failing hard as the x-component increases. This is more like case 1 than case 2, because gradient descent (as applied to reinforcement learning agents) doesn’t have access to expected reward - only to the reward actually received. So this will make no difference until the agent fails for the first time, after which it will bounce back and forth until something else changes. This seems to rule out gradient hacking via the method that Evan described, of having a hypothetical commitment that never actually needs to be implemented. But if the agent is repeatedly carrying out its commitment to fail, then there’ll be pretty strong pressure from gradient descent to change that. What changes might that pressure lead to? The two most salient options to me: The agent’s commitment to carrying out gradient hacking is reduced. The agent’s ability to notice changes implemented by gradient descent is reduced. Each of these makes the agent less likely to choose to fail at any given po...