Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] Existential Risk Analysis in Empirical Research Papers, published by Dan Hendrycks on July 2, 2022 on The AI Alignment Forum. I’ve noted before that it can be difficult to separate progress in safety from progress in capabilities. However, doing so is important, as we want to ensure that we are making differential progress on safety, rather than just advancing safety as a consequence of advancing capabilities. In particular, I think that research should rigorously evaluate trade-offs between improving safety and advancing capabilities. This paper introduces the concept of “capabilities externalities,” and provides concrete steps for how to analyze the risk of different research papers. It also introduces “x-risk sheets,” which can be added to papers and help researchers discuss their impact on AI x-risk. Since discussion of AGI is becoming increasingly normalized in mainstream ML, I think careful discussions of x-risk are increasingly feasible within mainstream ML. The paper is largely aimed at mainstream ML researchers, and it is one step towards normalizing the discussion of existential risks. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.