Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Defusing AGI Danger, published by Mark Xu on the AI Alignment Forum. This represents thinking about AGI safety done under mentorship by Evan Hubinger. Thanks also to Buck Shlegeris, Noa Nabeshima, Thomas Kwa, Sydney Von Arx and Jack Ryan for helpful discussion and comments. tl;dr A common perspective to take with respect to AI safety to try and construct a story in which artificial general intelligence (AGI) goes right. I have recently acquired another perspective that I find helpful, which is AGI safety as disrupting stories for why AGI will go poorly. I will start by presenting an analogy to convey intuition, then apply this perspective to concrete examples of current alignment strategies. Introduction Suppose you have a bomb that is about to explode. Your task is to make sure no one gets hurt. How might you reason about how to do this? One way to do this is , or backchaining, which involves identifying a sub-goal that, if achieved, will accomplish your main goal. This process is then recursively applied to generate a concrete next action. Applied to the situation at hand, I might identify that defusing the bomb would prevent everyone from getting hurt. However, I do not know how to defuse a bomb. A sub-goal that would help me defuse a bomb is learning about bombs, which might suggest that I take a class on bomb-defusal. This is not helpful because I do not have time to take such a class. In general, backchaining might fail when there are many possible ways to accomplish a goal because it might cause you to fixate on one possible path. Suppose that instead I ask “why do I think the bomb explosion will go poorly?”. The answer I get seems much more useful: the bomb will throw lots of little bits everywhere and those bits will hit people. This gives me two possible subquestions to ask: “why will the bomb throw bits everywhere?” and “why will those bits hit people?”. Answering the second question: the people are close by and there is nothing stopping the bits. Again, this yields two possible recursive subquestions. Notice how this framing forces me to consider many possible paths towards solving my goal simultaneously. The point of the second perspective is to construct a specific story for why the bomb explosion will hurt people. The goal is to eventually arrive at something that looks like “the timer will hit zero, which will activate this diode, allowing electricity to flow from the battery into the explosive. This will cause the bomb to explode, flinging bits of metal at high velocities. Since there are people close to the bomb and there is nothing in the way, these bits of metal will hit the people, hurting them”. If the story is complete, if it includes the full reason why the current situation might result in people getting hurt, then violating any step of the story should make the current situation safe. In this case, one is left with a number of possible solutions: Make the timer not hit zero Make the diode never activate Break the circuit, now allowing electricity to flow into the explosive Move the people farther away Put something in-between the bomb and the people Make the people bits-of-metal-proof Asking for a story of danger is a mental move that encourages specificity instead of abstraction. It is easier to be specific about something you expect to happen as opposed to conditioning on things that might happen and searching for explanations. Once one has generated a specific story of danger, the only remaining task is to violate one of its assumptions. Applied to AGI Safety You think that there is something called “AGI” that is going to cause bad things to happen. Your task is to make bad things not happen. How might you start reasoning about this? Instead of backchaining, we can try to disrupt the current forward chain that leads to a bad outcome. Inst...