Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Open Agency Architecture for Safe Transformative AI, published by davidad (David A. Dalrymple) on December 20, 2022 on The AI Alignment Forum.Note: my quality bar here is "this is probably worth the reader's time" rather than "this is as clear, compelling, and comprehensive as I can make it."In the Neorealist Success Model, I asked:What would be the best strategy for building an AI system that helps us ethically end the acute risk period without creating its own catastrophic risks that would be worse than the status quo?This post is a first pass at communicating my current answer.Bird's-eye viewAt the top level, it centres on a separation betweenlearning a world-model from data and eliciting desirabilities within that ontologyplanning against a world-model and associated desirabilitiesacting in real-timeWe see such a separation in, for example, MuZero, which can probably still beat GPT-4 at Go—the most effective capabilities do not always emerge from a fully black-box, end-to-end, generic pre-trained policy.HypothesesScientific Sufficiency Hypothesis: It's feasible to train a purely descriptive/predictive infra-Bayesian world-model that specifies enough critical dynamics accurately enough to end the acute risk period, such that this world-model is also fully understood by a collection of humans (in the sense of "understood" that existing human science is).MuZero does not train its world-model for any form of interpretability, so this hypothesis is more speculative.However, I find Scientific Sufficiency much more plausible than the tractability of eliciting latent knowledge from an end-to-end policy.It's worth noting there is quite a bit of overlap in relevant research directions, e.g.pinpointing gaps between the current human-intelligible ontology and the machine-learned ontology, andinvestigating natural abstractions theoretically and empirically.Deontic Sufficiency Hypothesis: There exists a human-understandable set of features of finite trajectories in such a world-model, taking values in (−∞,0], such that we can be reasonably confident that all these features being near 0 implies high probability of existential safety, and such that saturating them at 0 is feasible with high probability, using scientifically-accessible technologies.I am optimistic about this largely because of recent progress toward formalizing a natural abstraction of boundaries by Critch and Garrabrant. I find it quite plausible that there is some natural abstraction property Q of world-model trajectories that lies somewhere strictly within the vast moral gulf ofModel-Checking Feasibility Hypothesis: It could become feasible to train RL policies such that a formally verified, infra-Bayesian, symbolic model-checking algorithm can establish high-confidence bounds on its performance relative to the world-model and safety desiderata, by using highly capable AI heuristics that can only affect the checker's computational cost and not its correctness—soon enough that switching to this strategy would be a strong Pareto improvement for an implementation-adequate coalition.Time-Bounded Optimization Thesis: RL settings can be time-bounded such that high-performing agents avoid lock-in.I'm pretty confident of this.The founding coalition might set the time bound for the top-level policy to some number of decades, balancing the potential harms of certain kinds of lock-in for that period against the timelines for solving a more ambitious form of AI alignment.If those hypotheses are true, I think this is a plan that would work. I also think they are all quite plausible (especially relative to the assumptions that underly other long-term AI safety hopes)—and that if any one of them fails, they would fail in a way that is detectable before deployment, making an attempt to execute the p...