Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: PreDCA: vanessa kosoy's alignment protocol, published by Tamsin Leake on August 20, 2022 on The AI Alignment Forum. (this post has been written for the second Refine blog post day, at the end of the first week of iterating on ideas and concretely aiming at the alignment problem. thanks to vanessa kosoy, adam shimi, sid black, artaxerxes, and paul bricman for their feedback.) in this post, i try to give an overview of vanessa kosoy's new alignment protocol, Precursor Detection, Classification and Assistance or PreDCA, as she describes it in a recent youtube talk. keep in mind that i'm not her and i could totally be misunderstanding her video or misfocusing on what the important parts are supposed to be. the gist of it is: the goal of the AI should be to assist the user by picking policies which maximize the user's utility function. to that end, we characterize what makes an agent and its utility function, then detect agents which could potentially be the user by looking for precursors to the AI, and finally we select a subset of those which likely contains the user. all of this is enabled by infra-bayesian physicalism, which allows the AI to reason about what the world is like and what the results of computations are. the rest of this post is largely a collection of mathematical formulas (or informal suggestions) defining those concepts and tying them together. an important aspect of PreDCA is that the mathematical formalisms are theoretical ones to be given to the AI as-is, not specifications as to what algorithms or data structures should exist inside the AI. ideally, the AI could just figure out what it needs to know about them, to what degree of certainty, and using what computations. the various pieces of PreDCA are described below. Φ is the set of hypotheses about how the physical world could be — for example, different hypotheses could entail different truthfulness for statements like "electrons are lighter than protons" or "norway has a larger population than china". Γ is the set of hypotheses about what the outputs of all programs are — for example, a given hypothesis could contain a statement such as "2+2 gives 4", "2+2 gives 5", "the billionth digit of π is 7", or "a search for proofs that either P=NP or P≠NP would find that P≠NP". note that, as the "2+2 gives 5" example demonstrates, these don't have to be correct hypotheses; in fact, PreDCA relies a lot on entertaining counterfactual hypotheses about the results of programs. a given hypothesis γ∈Γ would have type γ : program output. Φ×Γ is the set of pairs of hypotheses — in each pair, one hypothesis about the physical world and one hypothesis about computations. note that a given hypothesis φ∈Φ or γ∈Γ is not a single statement about the world or computationspace, but rather entire descriptions of those. a given φ∈Φ would say everything there is to say about the world, and a given γ would specify the output of all possible programs. they are not to be stored inside the AI in their entirety of course; the AI would simply make increasingly informed guesses as to what correct hypotheses would entail, given how they are defined. □(Φ×Γ) assigns degrees of beliefs to those various hypotheses; in infra-bayesianism, those degrees are represented as "infra-distributions". i'm not clear on what those look like exactly, and a full explanation of infra-bayesianism is outside the scope of this post, but i gather that — as opposed to scalar bayesian probabilities — they're meant to encode not just the probability but also uncertainty about said probability. Θ is one such infra-bayesian distribution. vanessa emphasizes that infra-bayesian physicalist hypotheses are described "from a bird's eye view" as opposed to being agent-centric, which helps with embedded agency: the AI has guesses as to what the whole world is lik...