Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Overview of the AI Alignment Landscape: Threat Models, published by Neel Nanda on December 25, 2021 on The AI Alignment Forum. This is the second post in a sequence mapping out the AI Alignment research landscape. The sequence can be read in any order, though I recommend starting with the first post for a high-level summary of the sequence and how it all fits together. Disclaimer: I recently started as an interpretability researcher at Anthropic, but I wrote this post before starting, and it entirely represents my personal views not those of my employer Intended audience: People who understand why you might think that AI Alignment is important, but want to understand what AI researchers actually do and why. Pedagogy note: I link to many papers and blog posts to read more about each area. I think technical writing is often harder to digest without a big picture in mind, so where possible I link to Alignment Newsletter summaries for a piece. There are a lot of links, so I recommend reading the summaries for anything interesting, but being selective about which full-length works you read. Terminology note: There is a lot of disagreement about what “intelligence”, “human-level”, “transformative” or AGI even means. For simplicity, I will use AGI as a catch-all term for ‘the kind of powerful AI that we care about’. If you find this unsatisfyingly vague, OpenPhil’s definition of Transformative AI is my favourite precise definition. Introduction A common approach when setting research agendas in AI Alignment is to be specific, and focus on a threat model. That is, to extrapolate from current work in AI and our theoretical understanding of what to expect, to come up with specific stories for how AGI could cause an existential catastrophe. And then to identify specific problems in current or future AI systems that make these failure modes more likely to happen, and try to solve them now. It is obviously really hard to reason about the future in a specific way without being wildly off! But I am pretty excited about approaches like this. I think it's easy for research (or anything, really) to be meandering, undirected and not very useful, especially for vague and ungrounded problems such as AI Alignment, which is essentially trying to fix problems in a technology that doesn’t exist yet. And having a specific story to guide what you do can be a valuable source of direction, even if ultimately you know it will be flawed in many ways. Nate Soares makes the case for having a specific but flawed story in general well. Note that I think there is very much a spectrum between this category and robustly good approaches (a forthcoming post in this sequence). Most robustly good ways to help also address specific threat models, and many ways to address specific threat models feel useful even if that specific threat model is wrong. But I find this a helpful distinction to keep in mind. Pedagogy notes: When discussing threat models, it’s often helpful to give a specific story of exactly how things could go wrong. But this can be misleading, because we often find stories more compelling the more detailed they are, yet mathematically every time you add an extra detail to a story, it becomes less likely (often called the conjunction fallacy). As such, where possible, I try to distill each threat model down to a simple set of assumptions Often, the part I consider most interesting is less the specific threat model, and more the intuitions and worldviews that underlie it. As such, the descriptions of each case are often far longer than necessary, so I can flesh out my intuitions, give illustrative examples, etc. You can disagree with the threat model and agree with the intuitions, and vice versa Feel free to skip around if the sections feel overly long-winded, the high-level sections can be rea...