Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The longtermist AI governance landscape: a basic overview, published by SamClarke on January 18, 2022 on The Effective Altruism Forum. Aim: to give a basic overview of what is going on in longtermist AI governance. Audience: people who have limited familiarity with longtermist AI governance and want to understand it better. I don’t expect this to be helpful for those who already have familiarity with the field. This post outlines the different kinds of work happening in longtermist AI governance. For each kind of work, I’ll explain it, give examples, sketch some stories for how it could have a positive impact, and list the actors I’m aware of who are currently working on it.[1] Firstly, some definitions: AI governance means bringing about local and global norms, policies, laws, processes, politics, and institutions (not just governments) that will affect social outcomes from the development and deployment of AI systems.[2] Longtermist AI governance, in particular, is the subset of this work that is motivated by a concern for the very long-term impacts of AI. This overlaps significantly with work aiming to govern transformative AI (TAI). It’s worth noting that the field of longtermist AI governance is very small. I’d guess that there are around 60 people working in AI governance who are motivated by a concern for very long-term impacts. Short summary On a high level, I find it helpful to consider there being a spectrum between foundational and applied work. On the foundational end, there’s strategy research, which aims to identify good high-level goals for longtermist AI governance; then there’s tactics research which aims to identify plans that will help achieve those high-level goals. Moving towards the applied end, there’s policy development work that takes this research and translates it into concrete policies; work that advocates for those policies to be implemented, and finally the actual implementation of those policies (by e.g. civil servants). There’s also field-building work (which doesn’t clearly fit on the spectrum). Rather than contributing directly to the problem, this work aims to build a field of people who are doing valuable work on it. Of course, this classification is a simplification and not all work will fit neatly into a single category. You might think that insights mostly flow from the more foundational to the more applied end of the spectrum, but it’s also important that research is sensitive to policy concerns, e.g. considering how likely your research is to inform a policy proposal that is politically feasible. We’ll now go through each of these kinds of work in more detail. Research Strategy research Longtermist AI strategy research ultimately aims to identify high-level goals we could pursue that, if achieved, would clearly increase the odds of eventual good outcomes from advanced AI, from a longtermist perspective (following Muehlhauser, I’ll sometimes refer to this aim as ‘getting strategic clarity’). This research can itself vary on a spectrum between targeted and exploratory as follows: Targeted strategy research answers questions which shed light on some other specific, important, known question e.g. “I want to find out how much compute the human brain uses, because this will help me answer the question of when TAI will be developed (which affects what high-level goals we should pursue)” Exploratory strategy research answers questions without a very precise sense of what other important questions they’ll help us answer e.g. “I want to find out what China’s industrial policy is like, because this will probably help me answer a bunch of important strategic questions, although I don't know precisely which ones” Examples Work on TAI forecasting, e.g. biological anchors and scaling laws for neural language models. Example of strategic rele...