Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Intermittent Distillations #3, published by Mark Xuon the AI Alignment Forum. Mundane solutions to exotic problems (Paul Christiano) Mundane solutions to exotic problems Summary Thinking about AI safety often leads to considering exotic problems: models purposefully altering their gradients, agents hiding their capabilities to defect when an opportunity arises, or humans being vulnerable to side-channel attacks. These exotic problems might seem like they require exotic solutions. However, the author points out that the origin of these exotic problems is the models themselves having exotic capabilities. If one is able to train a model to use whatever capabilities it has for a good purpose, then if the model gains exotic capabilities, it'll also be able to use those to avoid exotic problems. As an analogy, if one hires a mercenary to help them when a war, they might be worried that the mercenary develops some weapon that means they are no longer motivated by your money. Since the mercenary has different goals than their employer, the employer must be worried about the incentive structure that they set up being broken. However, if one has a soldier that is fundamentally loyal, one is not at all worried about this soldier developing a powerful weapon, since their loyalty ensures they'll use this new exotic capability in service of your goals. If it turns out that the weapon requires special containment procedures, the soldiers' loyalty will ensure that they'll use their weapons expertise to help you contain it. Opinion This framing on capabilities redirecting is similar to framings provided in The strategy-stealing assumption and Universaliity Unwrapped. Empirically, it's taken me a while to understand the author's framing of "build an overseer that 'knows' everything the model knows" as a sufficient solution to the alignment problem, but I think it makes sense to me now. However, I still don't really understand why the author thinks this is a tractable problem. To be fair, I'm not sure why people think value learning is tractable either. I'd blame this on my lack of understanding more than anything. Elsewhere, the author has said their goal is to provide a clean solution to the alignment problem. I think this post gives intuition for why the messiness that sometimes appears at the surface of AI safety problems might be solvable with some deep solution. Low-stakes alignment (Paul Christiano) Low-stakes alignment Summary A model is operating in a low-stakes setting if the potential negative impact of any of its decisions are bounded at some pretty low amount. In particular, this implies that we only care about the long-run average behavior of the model. For example, a standard concern in AI safety is that the model seizes control of its reward channel. If it was possible to do this quickly, the model would have taken an action with large negative-impact, so it wouldn't be in a low-stakes setting. In particular, one is not concerned with distributional shift in low-stakes settings. Since each particular decision only has potentially small negative impact, we can simply train our model in an online setting, suffering from bad performance for a while until the model eventually learns the new distribution. If the individual decisions don't matter, then the overall cost of this transition period is low. Therefore, in low-stakes settings, the primary concern is that of outer alignment. If one has an objective that is both easy to optimize and will produce good behavior in the limit of optimization, training the model in a low-stakes setting will produce an aligned model without catastrophe (assuming the model is powerful enough to learn the objective). To see why this is the case, having an outer-aligned objective means the model is aligned in the limit. Since the setting is lo...