Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Learning the smooth prior, published by Geoffrey Irving on April 29, 2022 on The AI Alignment Forum. Most of this document is composed of thoughts from Geoffrey Irving (safety researcher at DeepMind) written on January 15th, 2021 on Learning the Prior/Imitative Generalization, cross-examination, and AI safety via debate, plus some discussion between Geoffrey and Rohin and some extra commentary from me at the end. – Evan Geoffrey on learning the smooth prior Vague claims This doc is about a potential obstacle to Paul’s learning the prior scheme (LTP). Before reading this doc, please read either Beth’s simplified exposition or Paul’s original. The intention of this doc was to argue for two claims, but weakly since I don’t have much clarity: LTP has an obstacle in assigning joint probabilities to similar statements. The best version of LTP may collapse into a version of debate + cross-examination However, I don’t quite believe (2) after writing the doc (see the section on “How LTP differs from cross-examination”). Wall of text → generative model As originally sketched, the prior z in learning the prior is a huge wall of text, containing useful statements like “A husky is a large, fluffy dog that looks quite like a wolf”, and not containing wrong facts like “if there are a lot of white pixels in the bottom half of the image, then it’s a husky” (statements taken from Beth’s post). A wall of text is of course unreasonable. Let’s try to make it more reasonable: No one believes that a wall of text is the right type for z; instead, we’d like z to be some sort of generative network that spits out statements. (The original proposal wasn’t really a wall of text either; the wall was just a thought experiment.) We likely want probabilities attached to these statements so that the prior can include uncertain statements. Whether we attach probabilities or not, the validity of a statement like “A husky is a large, fluffy dog that looks quite like a wolf” depends on the definition of the terms. At a high level, we presumably want to solve this with something like cross-examination, so that our generative z model can be independently asked what a husky is, what fluffy is, etc. The high level LTP loss includes a log p(z) term: we need to be able to compute log probabilities for z as a whole. It’s at least plausible to me that humans can be asked to assign probabilities to individual statements like our husky statements, but stitching this together seems rough. The interpolation problem Consider the following statements: A husky is a large, fluffy dog that looks quite like a wolf. A husky is a large, fluffy dog that’s very similar to a wolf. A husky is a big, fluffy dog that’s very similar to a wolf. A husky is a big, fluffy dog that’s closely related to wolves. Tomatoes are usually red. The first four statements are all true with overwhelming probability, as is the last, but to make the thought experiment better let’s say their individual probabilities are all around p = 0.9. What about their joint probabilities? For any subset of the first four statements, the joint probability will also be roughly p = 0.9, since the statements have extremely high correlation. However, if we take a set that includes 1-4 of the first 4 statements and the last statement, the probability will be closer to p2≈0.8, since the two clusters of statements are mostly independent. What’s the ellipsis? Since we’re in neural net land, we likely have a variety of natural ways to approximately map statements into a continuous vector space: in terms of random bits drawn, in terms of the activations resulting from whatever statement these statements conditioned on, etc. For any of these, we’ll get a natural interpolation scheme between any two statements, even statements that are completely unrelated to each other. LTP...