Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: It matters when the first sharp left turn happens, published by Adam Jermyn on September 29, 2022 on The AI Alignment Forum. Thanks to Evan Hubinger for comments on these ideas. Introduction A “sharp left turn” is a point where capabilities generalize beyond alignment. In a sharp left turn an AI becomes much more capable than aligned, and so starts to exploit flaws in its alignment. This can look like Goodharting, where strong optimization pressure causes outer alignment failure because the base goal isn’t identical to what we want. Or it can look like deceptive alignment, where the model is aligned to proxy goals that aren’t identical to the base goal, but we don’t notice until the model is capable enough to make the proxies fail. However they happen, sharp left turns are bad. An important question, though, is: when will the first sharp left turn happen? Timing Consider three scenarios: Weak: The sharp left turn happens before models are able to cause existential risks. For instance, maybe GPT-4 has a sharp left turn but isn’t able to reliably execute on its plans. Strong: The sharp left turn happens once capabilities are very super-human and dangerous if misused. In this world AI’s are doing the bulk of the world’s scientific research by the time alignment unravels. Human-level: The sharp left turn happens while capabilities are around human level, meaning both dangerous and not game-changingly helpful for research. I think that the most dangerous of these is the human-level world. In the “weak” scenario, we get to see the sharp left turn coming. We potentially get lots of experience with models looking aligned and then that alignment failing with increasing scale. In this world, we effectively get multiple shots at alignment because the first badly-misaligned models just aren’t that dangerous. We get to experiment with deceptively aligned models and get a grounded empirical understanding of why models become deceptively aligned and how we might stop that. In this world, alignment research gets a chance to become a paradigmatic scientific field before the models get too capable. In the “strong” scenario, we get to do alignment research with super-human models that are still aligned. In this world, we don’t have to solve the technical challenge of alignment entirely on our own. Alignment research potentially gets much further along before we have to have an airtight solution and prevent sharp left turns. Both of these seem like pretty hopeful worlds. By contrast, the world where sharp left turns happen near human-level capabilities seems pretty dangerous. In that world, we don’t get to study things like deceptive alignment empirically. We don’t get to leverage powerful AI researcher tools. We have to solve the problem entirely on our own, and we have to do it without any useful empirical feedback. This seems really bad. Scales The timing of sharp left turns depends on a few capability scales, some of which I conflated above: The danger scale, above which the model is capable enough to pose an existential risk. The research scale, above which the model is able to do scientific research better than humans. The left-turn scale, above which the model is able to undergo a sharp left turn. A useful anchor here is given by deceptive alignment, which suggests the scale at which a model can successfully reason about its own training process. The three scenarios I outlined are characterized by different orderings of these scales: Weak: (left-turn < danger) Strong: (research < left-turn) Human-level: (left-turn ~ research, left-turn ~ danger) There can be other orderings, leading to other outcomes. For instance, we could live in a world with (danger < left-turn < research). In that world, the left turn happens at a scale that’s dangerous but incapable of super-human research. ...