Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Endgame Stories, published by Ivan Vendrov on September 28, 2022 on The AI Alignment Forum. Assume you are in the set of possible worlds where AI takeover happens by default. If you do nothing, then at some point in the 21st century the AI lab Magma develops a transformative AI system. Magma employees perform a number of safety checks, conclude the system is safe enough, and deploy it. They deploy it slowly and incrementally, with careful monitoring. But despite their efforts, the system turns out to be unsafe and the monitoring insufficient, triggering a cascade of events eventually leading to an existential catastrophe. I’ll refer to this sequence of events as the “baseline story” going forward. Assume further that you’re in the narrower set of worlds where this AI catastrophe is contingent on your actions. In other words, there exists a sequence of actions you (or your organization) can take that averts catastrophe, a decisive intervention. Not necessarily a pivotal act, an intervention that averts all existential risk from AI. Just an intervention that prevents this specific Magma catastrophe, giving humanity some breathing room, perhaps only a few months or years. Let’s try to understand what this decisive sequence of actions could look like. It’s tempting to start at the beginning of the sequence and think about what the first few actions look like. Unfortunately, the most probable starting actions are “meta” actions like thinking really hard, talking to experts, or recruiting more people to work on the problem. These are the same kinds of actions that any successful project starts with! So it doesn’t help us constrain the space of decisive interventions. Instead, it’s more helpful to start with the endgame: how, in the end, did your actions change the baseline story and avert catastrophe? And what were the last nodes in the causal chain leading up to the change? At the most abstract level, the baseline story has the following structure. A social process (Magma) instantiates a technological artifact (unsafe AI) which destroys the world. There are three objects here, and any change to the story requires changing the properties of at least one of them. This leads naturally to a 3-way categorization of endgame stories, though the real endgame story will probably involve some combination of the three: Changing the technology. You changed something about the technology that Magma had access to, which delayed the deployment or made it safe. Changing the decision maker. You changed something about Magma, or more broadly the social decision process that led to the unsafe deployment. Changing the broader world. You changed something about the broader world that made it resilient to Magma’s deployment decision. In the rest of this post I'll use this categorization to walk through a diverse array of endgame stories. Changing the Technology Differential Development of Safety Let’s start with the broad endgame story that most technical alignment research is aimed at, differential development of safety technology (a special case of differential technological development): You develop technology that makes AI safer, with mild competitiveness penalties. Because of your direct work, the technology is developed N years faster than it would have been by default. Magma’s deployment falls within that N-year window, so they use your technology, so their deployment does not lead to catastrophe. Can we make this story more concrete? How exactly does the new technology prevent catastrophe? The simplest path involves finding decisive safety augmentation, something like “add an adversarial robustness term to the training objective” - a conceptual breakthrough that Magma adopts immediately because it is relatively easy to implement and aligned with business objectives. Another s...