Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sticky goals: a concrete experiment for understanding deceptive alignment, published by Evan Hubinger on September 2, 2022 on The AI Alignment Forum. Thanks to Sam McCandlish for a useful conversation that prompted this post. In “How likely is deceptive alignment?”, I lay out the case for why you might expect deceptive alignment to occur in practice for two different stories of ML inductive biases: high and low path-dependence. In this post, I want to propose an experiment that I think would shed substantial light on the high path-dependence case specifically. In the high path-dependence case, the story for why you would get deceptive alignment is essentially as follows: The model learns some set of proxy goals. The model learns enough about the training process that, if it were to use that knowledge to directly optimize for what the training process is trying to get it to do, it would get better performance than just using its proxy goals. Gradient descent modifies the model’s proxies to become long-term goals, making the model deceptive such that it starts optimizing directly for its understanding of what the training process wants and thus gets better performance. However, as I talk about in “How likely is deceptive alignment?”, there is another, alternative option available to gradient descent for step (3): instead of turning the existing proxies into long-term goals, gradient descent could just replace them entirely with a pointer to the model’s understanding of the training objective. In my opinion, I think it is currently quite unclear which one of these gradient descent would prefer. One way to think about these two options is that gradient descent is effectively choosing between: repurposing the existing terminal goal to care about the training objective instrumentally vs. modifying the existing terminal goal to care about the training objective terminally. Thus, we can conceptualize the question of which would be most likely to happen as: how “sticky” are an agent’s existing terminal proxies? Now, one thing I will say is that, since the difficulty of changing the proxies into long-term goals seems to be essentially constant regardless of the complexity of the training objective—whereas the difficulty of creating a pointer to the training objective scales with that complexity—I think that, if we try to train models on increasingly complex goals, eventually deception will win almost regardless of the baseline “stickiness” level. But understanding that baseline could still be quite important, and it’s something that I think we can quite readily experiment with right now. Furthermore, understanding of goal stickiness could also help shed additional light on the broader question of whether machine learning inductive biases look more like the high or low path-dependence world to begin with. If we see very little goal stickiness, that might just be because essentially nothing is sticky and we shouldn’t be conceptualizing anything as path-dependent to begin with. Here’s an example experimental setup for trying to understand goal stickiness: Train an agent (via RL, via a decision transformer, etc.) to accomplish some complex goal in a simulated environment requiring intermediate resources. Concrete example: Train an RL agent to build diamond houses in a particular Minecraft chunk. Fine-tune in an environment with two distinct stages. In stage 1, fine-tune the agent to get as much of the intermediate resource as possible and prevent the agent from being able to achieve the complex goal. In stage 2, provide no rewards (except maybe something like next state prediction) and allow the agent to do whatever it wants. Iterate. Concrete example: Fine-tune the agent to get as many diamonds as possible while being locked out of the special chunk in stage 1, then let it do wha...