Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Towards formalizing universality, published by Paul Christiano on the AI Alignment Forum. (Cross-posted at ai-alignment.com) The scalability of iterated amplification or debate seems to depend on whether large enough teams of humans can carry out arbitrarily complicated reasoning. Are these schemes “universal,” or are there kinds of reasoning that work but which humans fundamentally can’t understand? This post defines the concept of “ascription universality,” which tries to capture the property that a question-answering system A is better-informed than any particular simpler computation C. These parallel posts explain why I believe that the alignment of iterated amplification largely depends on whether HCH is ascription universal. Ultimately I think that the “right” definition will be closely tied to the use we want to make of it, and so we should be refining this definition in parallel with exploring its applications. I’m using the awkward term “ascription universality” partly to explicitly flag that this is a preliminary definition, and partly to reserve linguistic space for the better definitions that I’m optimistic will follow. (Thanks to Geoffrey Irving for discussions about many of the ideas in this post.) I. Definition We will try to define what it means for a question-answering system A to be “ascription universal.” 1. Ascribing beliefs to A Fix a language (e.g. English with arbitrarily big compound terms) in which we can represent questions and answers. To ascribe beliefs to A, we ask it. If A(“are there infinitely many twin primes?”) = “probably, though it’s hard to be sure” then we ascribe that belief about twin primes to A. This is not a general way of ascribing “belief.” This procedure wouldn’t capture the beliefs of a native Spanish speaker, or for someone who wasn’t answering questions honestly. But it can give us a sufficient condition, and is particularly useful for someone who wants to use A as part of an alignment scheme. Even in this “straightforward” procedure there is a lot of subtlety. In some cases there are questions that we can’t articulate in our language, but which (when combined with A’s other beliefs) have consequences that we can articulate. In this case, we can infer something about A’s beliefs from its answers to the questions that we can articulate. 2. Ascribing beliefs to arbitrary computations We are interested in whether A “can understand everything that could be understood by someone.” To clarify this, we need to be more precise about what we mean by “could be understood by someone.” This will be the most informal step in this post. (Not that any of it is very formal!) We can imagine various ways of ascribing beliefs to an arbitrary computation C. For example: We can give C questions in a particular encoding and assume its answers reflect its beliefs. We can either use those answers directly to infer C’s beliefs (as in the last section), or we can ask what set of beliefs about latent facts would explain C’s answers. We can view C as optimizing something and ask what set of beliefs rationalize that optimization. For example, we can give C a chess board as input, see what move it produces, assume it is trying to win, and infer what it must believe. We might conclude that C believes a particular line of play will be won by black, or that C believes general heuristics like “a pawn is worth 3 tempi,” or so on. We can reason about how C’s behavior depends on facts about the world, and ask what state of the world is determined by its current behavior. For example, we can observe that C(113327) = 1 but that C(113327) “would have been” 0 if 113327 had been composite, concluding that C(11327) “knows” that 113327 is prime. We can extend to probabilistic beliefs, e.g. if C(113327) “probably” would have been 0 if 113327 had been composite, th...