Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some thoughts about imperfect world modeling, published by leogao on April 7, 2022 on The AI Alignment Forum. So a few posts ago I looked at the problem of not being able to anticipate all consequences of an action as being related to deceptive mesaoptimization, but also outer alignment too. This post digs more into some of the things I only touched on briefly in that post. Editor’s note: I’m experimenting with having a lower quality threshold for just posting things even while I’m still confused and unconfident about my conclusions, but with this disclaimer at the top. Thanks to Kyle and AI_WAIFU for discussions. Last time, we narrowed down the problem to [certain actions we can take now] leading to [a world such that if [RSA2048 or some other thing we can’t simulate] happens, then [bad things will happen]] (brackets added for easier parsing of this sentence). This is because we can't plan for a future where RSA-2048 is solved but if we can already see RSA2048 in the world, then we can plan forward from that and see when things blow up. So if we could somehow figure out how to ensure that no action puts us into an RSA2048-vulnerable world, then we can prevent this failure case. Unfortunately, knowing precisely whether a world is vulnerable would essentially require us to simulate RSA2048, which gets us nowhere. However, one observation is we don't have to know precisely, we just need to know whether a world could be vulnerable — like for example if the agent removes its own safeguard, then we wouldn’t want to allow that even if the model is in fact safe and wouldn’t destroy the world when it sees RSA2048. Telling whether side effects are dangerous is hard One thing we might think to do is instead simulate putting a dummy AI in the “do rollouts to find out if these actions destroy the world” box (the word “box” here has no relation to the AI box experiment), such that this dummy AI tries to kill everyone when it sees something we can actually simulate, like the string "chariots chariots" or something. Then, we can execute the action in the simulation, and then simulate seeing that string, and see if the box successfully prevented the action from being executed in the real world. This way, we can test whether the box is broken by the action, right? Unfortunately there's a huge counterexample to this. So suppose the AI proposes building a really complicated machine in the world that we don't understand. We can still simulate the consequences of having this machine exist, of course, but we can't simulate it past the RSA2048 barrier, and so this machine could do evil stuff after seeing RSA2048 and we'd never know. The more general lesson from this is that if the model proposes doing something with side effects in the real world that we can't understand, that allows it to build up stuff "outside of the box" and then that stuff could do bad things. If we try to understand everything the model does, we end up bottlenecked by the whole "doing science" thing again, and if we try to bring the things the model does back inside the box (i.e only allow the machine it builds to also propose actions within the box), then you end up never being able to modify the real world at all because any changes to the real world could actually be setting up something that blows up after RSA2048, which is totally useless. So, in general, it seems hard to know whether an action sets up something evil in the world that only activates after RSA2048. It’s (partial) ELK reduction time, my dudes There is a partial ELK reduction here in that if we can solve ELK then we can ask the model whether it thinks its actions are putting the world into a vulnerable state. It's worth noting that this wouldn't work if we just asked the model whether RSA2048 would cause it to kill everyone, because a) the model co...