Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Refining the Sharp Left Turn threat model, part 2: applying alignment techniques, published by Victoria Krakovna on November 25, 2022 on The AI Alignment Forum. A Sharp Left Turn (SLT) is a rapid increase in model capabilities (such as planning and world modeling). In a previous post, we broke down the SLT threat model into 3 claims: Capabilities will generalize far (i.e. to many domains) Alignment techniques that worked previously will fail during this transition Humans can’t intervene to prevent or align this transition We then proposed some possible mechanisms for Claim 1. This follow-up post will investigate possible arguments and mechanisms for Claim 2. In particular, we will outline our current understanding of the most promising plan for getting through an SLT and how it could fail (conditional on an SLT occurring). Plan: we use alignment techniques to find a goal-aligned model before SLT occurs, and the model preserves its goals during the SLT. We can try to learn a goal-aligned model before SLT occurs: a model that has beneficial goals and is able to reason about its own goals. This requires the model to have two properties: goal-directedness towards beneficial goals, and situational awareness (which enables the model to reason about its goals). Here we use the term "goal-directedness" in a weak sense (that includes humans and allows incoherent preferences) rather than a strong sense (that implies expected utility maximization). One can argue that the goal-aligned model has an incentive to preserve its goals, which would result in an aligned model after SLT. Since preserving alignment during SLT is largely outsourced to the model itself, arguments for alignment techniques failing during an SLT don't imply that the plan fails (e.g. it might be fine if interpretability or ELK techniques no longer work reliably during the transition if we can trust the model to manage the transition). Step 1: Finding a goal-aligned model before SLT We want to ensure that the model is goal-oriented with a beneficial goal and has situational awareness before SLT. It's important that the model acquires situational awareness at the right time: after it acquires beneficial goals. If situational awareness arises when the model has undesirable goals, this leads to deceptive alignment. Thus, our model search process would follow a decision tree along these lines: If situational awareness is detected without goal-directedness, restart the search. If undesirable goal-directedness or early signs of deceptive alignment are detected, restart the search. If an upcoming phase transition in capabilities is detected, and the model is not goal-aligned, restart the search. If beneficial goal-directedness is detected without situational awareness, train the model for situational awareness. Here are some ways alignment techniques could be applied in this plan: Use ELK techniques to identify whether the model has situational awareness (i.e. whether it believes it's inside a training process) and what goals it believes itself to have. Use interpretability techniques to pick up on early signs of deceptive alignment and restart and improve the search. (If we develop more advanced interpretability techniques that make it possible to retarget the internal search of the model, we can retarget a goal-directed model to a beneficial goal instead of restarting the search.) Use foundational progress on abstraction to determine whether the goals learned by the system are beneficial. Use foundational research insights (e.g. selection theorems) to make the search process more effective at avoiding models with undesirable goals and finding goal-aligned models Use insights from studying capability transitions to predict sharp increases in problematic capabilities, and make sure we only cross that threshold after we...