Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI X-risk >35% mostly based on a recent peer-reviewed argument, published by michaelcohen on November 2, 2022 on The AI Alignment Forum. My recent paper Advanced Artificial Agents Intervene in the Provision of Reward is a prerequisite to this essay (~5800 words; the section on the Assistance Game can be skipped). In that paper, I identify 6 assumptions from which it follows that a sufficiently advanced artificial agent planning over the long-term toward a goal in an unknown environment would cause everyone to die. This essay aims to establish that the probability of such an existential catastrophe is greater than 35% for the purpose of the Future Fund’s AI Worldview Prize. In the main part of this essay (~5700 words), I will divide up possible futures and assign credences to these outcomes. In particular, I will assign credences to each of the assumptions from my recent paper. Two appendices follow that can be skipped if necessary. In Appendix A (~7000 words), I review much-discussed proposals from the AI safety research community and explain why they do not solve the problem presented in the paper. In Appendix B (~1300 words), I extend the "Potential Approaches" section from my recent paper to add a class of approaches I hadn't recognized as potentially viable. I understand not all arguments will necessarily be read by the Future Fund, so let me briefly argue that this is worth the reader’s time. The antecedent paper is the only peer-reviewed argument that AI X-risk is likely. It has been certified as valid by professional computer scientists who are obviously not just deferring to conventional EA wisdom. My position cannot be accused of merely being tuned to the resonant frequency of an echo chamber. Finally, I am willing to bet up to $500 against someone else's $250 that: conditional on Nick Beckstead confirming he has read my recent paper and this essay, I will win prize money from the Future Fund. Section 1. Existentially dangerous levels of capability In my previous paper, I only make statements about "sufficiently advanced AI". What this means in clearer terms is AI that is capable enough to take over the world if it wanted to. Taking over the world appears to be hard. I do not claim and I do not expect that merely human-level AI presents any existential risk to us, except through their assistance in creating more advanced AI. Killing everyone requires being impervious to our attempts to shut it off once we notice something is wrong. If we are not able to destroy it, even just with conventional weaponry, that requires taking over our strategically relevant infrastructure, which is what I mean by taking over the world. I'll call an AI that is capable enough to present an existential risk "dangerously advanced AI". A note of interpretation: I understand the Future Fund to be asking about the probability that an AI-caused existential catastrophe ever happens conditional on AGI being deployed by 2070, not p(AI existential catastrophe by 2070 | AGI by 2070). So I do not suggest any upper limit on the time it may take to go from human-level AI to dangerously advanced AI. Section 2. Outline I will identify several possible routes to survival, and assign credences to each. All credences here will be optimistically biased, and they will still imply an X-risk >35%. To get the total chance of survival, I will add up the credences on each of the several routes, as if they are perfectly anti-correlated; this is an optimistic treatment of the correlation between these possibilities. Don't assume my true credences are close to the optimistic ones here. Or that if optimistic credence A > optimistic credence B, then true credence A > true credence B. The optimistic credences are just such that I am willing to spend some time defending them against a more optimistic objector. T...