Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ordering capability thresholds, published by Tamsin Leake on September 16, 2022 on The AI Alignment Forum. (this post has been written for the third Refine blog post day) given an AI which is improving towards ever more capabilities, such as by way of recursive self-improvement, in what order will it pass the following points? throughout this post i'll be using PreDCA as an example of a formal goal to be maximized, because it appears to me as a potentially promising direction; but you can imagine adapting this post to other formal goals such as insulated goal-programs, or other alignment strategies altogether. we can even use this time-ordering framework to compare the various thresholds of multiple alignment strategies, though i won't do that here. Start: we start the AI Math: it can figure out relatively complicated math, such as whether P equals PSPACE, or whether this world looks like it has finite compute if we can make it do physics. PreDCA: it can figure out what is entailed in maximizing PreDCA — notably that that goal is best entailed by not destroying the earth too much sub-PreDCA: it can figure out some individual parts of PreDCA, such as the identity of the user or what is entailed in maximizing a human's utility function, in a way that we can use to modify those parts if they need adjusting Escape: it becomes able to escape the environment over which we have control — and typically starts replicating across the internet Influence: it gets the ability to significantly influence the timeline, for example enough to eg save us from facebook destroying everything six months later DSA: it achieves decisive strategic advantage Doom: it becomes capable of destroying the earth too much (without necessarily using that capability) Cone: it takes over a significant portion of the universe, or at least of the lightcone with a few notes: "decisive strategic advantage" is a term i'm taking from bostrom's superintelligence book, describing the point at which an AI has sufficiently ensured its continuation that we can't turn it off or change its goals anymore; it is effectively the point of no return. by "destroying the earth too much" i mean destroying so much of earth that it can't reasonably be resimulated. if resimulating earth is too unethical, computationally costly, or anthropically costly, then "destroying too much of earth" might straightforwardly mean destroying all of humankind or something like that. note that for PreDCA, preserving earth in some way is important not just because it's pretty bad that we all die, but also because the AI might need to preserve its user and possibly their environment in order to figure out their utility function. in the case of knowing mathematical statements (Math, PreDCA, and sub-PreDCA), i imagine the AI being pretty sure about them, not necessarily having proven them. in addition, for simplicity, i'm assuming that we can use the AI to figure out some mathematical fact if and only if the AI can figure it out for itself — in practice, this need not be the case. one thing that can be noticed is that humans might serve as evidence. for example, we can examine history to figure out whether we passed Math or would've been able to pass PreDCA (given a reasonable description of it) before getting to Doom — my guess is yes at least for that latter one. now, we can reasonably guess the following pieces of ordering, where as usual in ordering graphs X Y means X < Y and transitive edges are not shown. in addition, for any two quantities X < Y, it can be the case that they're pretty close in time X ≈ Y, or it can be that there's a bunch of time between them X ≪ Y. whether the threshold between those two possibilities is more like a day or a year, is gonna depend on context. depending on how the rest of the ordering graph turns out and h...