Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I'm co-founding Aligned AI, published by Stuart Armstrong on February 17, 2022 on The AI Alignment Forum. I'm leaving the Future of Humanity Institute, the best and most impactful job I've ever had, to co-found Aligned AI. For the first time in my research career, I feel the problem of AI alignment is solvable. How? Alignment research: a history of partial failures The history of AI safety is littered with failures and partial successes. The most common examples of failure are ideas that would work well typically, but which fail in extreme situations - and a superintelligent AI is perfectly capable of creating such situations. Low-impact AIs were supposed to allow smart machines that interacted with humans without causing huge disruptions. They had some success at 'almost no impact'. But everyone - including me - failed at developing algorithms that had reliable low-impact. If the AI is allowed even a little bit of impact, it can make these low-impact restrictions irrelevant. Corrigibility and interruptibility were designed to allow AIs to be reprogrammed even when active and powerful. They have good narrow uses, but aren't a general solution: though the AI would not interfere with the interruptibility process, it also has no incentive to preserve it or to ensure its subagents were also interruptible. Oracles, question answering AIs (and their close relatives, tool AIs) are perennial suggestions, the idea being to limit the power of the AI by limiting it to answering questions or giving suggestions. But that fails, for instance when the AI is incentivised to manipulate humans through the contents of its answers or suggestions. There were some interesting examples on limiting AI power, but these were ultimately vulnerable to the AI creating subagents. The different forms of value learning confronted a surprising obstacle: values could not be learnt without making strong assumptions about human rationality, and human rationality could not be learnt without making strong assumptions about human values. A litany of partial failures suggests that the next approach tried will be a failure as well - unless we can identify why the approaches above failed. Is there a common failure mode for all of them? The common thread: lack of value extrapolations It is easy to point at current examples of agents with low (or high) impact, at safe (or dangerous) suggestions, at low (or high) powered behaviours. So we have in a sense the 'training sets' for defining low-impact/Oracles/low-powered AIs. It's extending these examples to the general situation that fails: definitions which cleanly divide the training set (whether produced by algorithms or humans) fail to extend to the general situation. Call this the 'value extrapolation problem[1], with 'value' interpreted broadly as a categorisation of situations into desirable and undesirable. Humans turn out to face similar problems. We have broadly defined preferences in familiar situations we have encountered in the world or in fiction. Yet, when confronted with situations far from these, we have to stop and figure out how our values might possibly extend[2]. Since these human values aren't - yet - defined, we can't directly input them into an algorithm, so AIs that can't solve value extrapolation can't be aligned with human values. Value extrapolation is thus necessary for AI alignment. It is also almost sufficient, since it allows AIs to draw correct conclusions from imperfectly defined human data. Combined with well grounded basic human values, it will allow the algorithm to extrapolate as well as humans can - better, in fact, using its superhuman abilities. If that's successful, AIs that value extrapolate and that start aligned, will remain aligned even as they dramatically change the world and confront the unexpected, re-assessing...