Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Embedding safety in ML development, published by Chin Ze Shen on October 31, 2022 on The AI Alignment Forum. This post was written as part of Refine. Thanks to Adam Shimi, Alexander Gietelink Oldenziel, and Vanessa Kosoy for helpful discussion and feedback. Summary This post aims to: Advocate for embedding safety into development of machine learning models Propose a framing on how to think about safety, where three factors contribute to an AI being dangerous, namely misalignment, optimization, and influence Discuss the pros and cons of this framing None of the ideas in this post are novel, and some of them may not even be practical, but they may be a useful framing to the problem of AI safety. Introduction Motivations There are many ways to get to AGI. It could be recursively self improving AI, or something more brain-like, or something that comes from improvements in machine learning systems, or something else much more alien to us. I don’t know which pathways are the ones our world will eventually end up in, but given recent rapid advances in ML recently, in this post I will address the scenario where the first dangerous AGI is achieved by mostly scaling up and improving upon present-day ML systems. In the absence of a robust solution to the hard problem of alignment, I think we should at least try to make the development of machine learning models go safely for as long as possible. This may be especially valuable if the first AGI we get is not yet an all-powerful superintelligent. It might be a fairly weak AGI capable of inflicting a lot of damage, but the damage could have been prevented. Not having a solution to the hard problem should not mean getting ourselves into preventable disasters. What safety means in this context Most safety fields in the real world don’t solve safety with guarantees. Whether it is flying an aircraft, operating a nuclear plant, or constructing a building, safety is all about effective risk management. This framing of safety has not been commonly used in the field of AI safety, for very good reasons, as the problem of a misaligned AGI is an unbounded one, where a ‘safety failure’ with a strong form of AGI could mean human extinction even with just a tiny degree of misalignment. For the purpose of this post, I am using the word “safety” in a more narrow sense than most people would. In terms of time-scale, I refer to “we are safe for now” rather than “things go great in the long run”; while in terms of failure mode, I refer to “we are safe from misaligned AIs” rather than “we are safe from AIs being used by bad actors and multipolar failures etc”. Framing the problem The question to be addressed is: what makes an AI harmful? I propose a framing that contains three ingredients: Misalignment at any level Optimization towards some goal Influence over the world The combination of optimization and influence can be loosely thought of as agency. Without any of the above ingredients, there will not be a catastrophe caused by a misaligned ‘agentic’ AI, as there would only be the following kinds of AI: Aligned: A fully aligned AI that has strong optimization and high influence over the world will do exactly what we want effectively, possibly leading us to utopia. Dormant: A misaligned AI that has a lot of influence over the world but not trying to do anything will probably not really be doing anything, like a deactivated nuclear bomb. Sponge: A misaligned AI that has a lot of optimization power but no influence over the world will not be doing anything in the real world, effectively being a sponge AI. I consider most of the powerful AIs today, such as AlphaGo Zero and DALL-E 2, as sponges for now, as they neither construct world-models nor have physical control over things in the physical world, but it is debatable if they will remain so in the fu...