Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nearcast-based "deployment problem" analysis, published by HoldenKarnofsky on September 21, 2022 on The AI Alignment Forum. When thinking about how to make the best of the most important century, two “problems” loom large in my mind: The AI alignment problem: how to build AI systems that perform as intended, and avoid a world run by misaligned AI. The AI deployment problem (briefly discussed here): the question of how and when to (attempt to) build and deploy powerful AI systems, under conditions of uncertainty about how safe they will be and how close others are to deploying powerful AI of their own. This piece is part of a series in which I discuss what both problems might look like under a nearcast: trying to answer key strategic questions about transformative AI, under the assumption that key events (e.g., the development of transformative AI) will happen in a world that is otherwise relatively similar to today's. A previous piece discussed the alignment problem; this one discusses the deployment problem. I’m using the scenario laid out in the previous post, in which a major AI company (“Magma,” following Ajeya’s terminology) has good reason to think that it can develop transformative AI very soon (within a year), using what Ajeya calls “human feedback on diverse tasks” (HFDT) - and has some time (more than 6 months, but less than 2 years1) to set up special measures to reduce the risks of misaligned AI before there’s much chance of someone else deploying transformative AI. I discuss what Magma would ideally do in this situation. I’m also introducing another hypothetical actor in this scenario, “IAIA2”: an organization, which could range from a private nonprofit to a treaty-backed international agency, that tracks3 transformative AI projects and takes actions to censure or shut down dangerous ones, as well as doing other things where a central, neutral body (as opposed to an AI company) can be especially useful. (More on IAIA below.) I’m going to discuss what Magma’s and IAIA’s major goals and priorities should be in the “nearcast” situation I’m contemplating; a future piece will go through what a few stylized success stories might look like. I’ll be bracketing discussion of the details of how Magma can reduce the risk that its own AI systems are misaligned (since I discussed that previously), and focusing instead on what Magma and IAIA should be looking to do before and after they achieve some level of confidence in Magma’s systems’ alignment. I focus on Magma and IAIA for concreteness and simplicity (not because I expect there to be only two important actors, but because my takes on what most actors should be doing can be mostly inferred from how I discuss these two). I sometimes give more detail on Magma, because IAIA is a bit more speculative and unlike actors that exist today. My discussion will be very high-level and abstract. It leaves a lot of room for variation in the details, and it doesn’t pin down how Magma and IAIA should prioritize between possible key activities - this is too sensitive to details of the situation. Nonetheless, I think this is more specific than previous discussions of the deployment problem, and for one who accepts this broad picture, it implies a number of things about what we should be doing today. I’ll discuss these briefly in the final section, and more in a future post. Summary of the post (bearing in mind that within the nearcast, I’m using present tense and not heavily flagging uncertainty): I’ll first give a bit more information on the hypothetical setting of this nearcast (specifically, on the addition of IAIA to the scenario discussed previously). I’ll break this scenario up into three stylized “phases,” even though in practice I think the boundaries between them could be fuzzy. “Phase 1” refers to the period of time when...