Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Speed + Simplicity Prior is probably anti-deceptive, published by Yonadav Shavit on April 27, 2022 on The AI Alignment Forum. Thanks to Evan Hubinger for the extensive conversations that this post is based on, and for reviewing a draft. This post is going to assume familiarity with mesa-optimization - for a good primer, check out Does SGD Produce Deceptive Misalignment by Mark Xu. Deceptive inner misalignment is the situation where the agent learns a misaligned mesaobjective (different from the base objective we humans wanted) and is sufficiently "situationally aware" to know that unless it deceives the training process by pretending to be aligned, gradient descent may alter its mesaobjective. There are two different reasons that an AI model could become a deceptive mesaoptimizer: During early training (before Situational Awareness), the agent learns a mesaobjective that will generalize poorly on the later-training/validation distribution. Once the mesaoptimizer becomes Situationally Aware, it will seek to actively avoid changes to whatever mesaobjective it had at that moment. I'll call this argument "path dependence". Alternatively, it may be that mesaoptimizer is misaligned even on the training distribution. Given sufficient optimization pressure, the learning process may favor a NN that is a mesaoptimizer with the simplest possible objective (which would fail to get any reward in the real environment), and that a misaligned objective of this sort can persist through deception alone. I'll call this argument "malign priors". In this post, I'll focus on the "malign priors" argument, and why I think a well-tuned speed prior can largely prevent it. Why does this matter? Well, if deceptive inner misalignment primarily occurs due to path dependence, that implies that ensuring inner alignment can be reduced to the problem of ensuring early-training inner alignment - which seems a lot more tractable, since this is before the model enters the "potentially-deceptive" regime. First, why would anyone think (2) was actually likely enough to justify studying it? I think the best reason is that by studying these pressures in the limit, we can learn lessons about the pressures that exist on the margin. For example, say we have an objective B that is perfectly-aligned on the training data, and there's a very-slightly-simpler objective B′ that is slightly worse on the training distribution. We might ask the question: is SGD likely to push B to become B′, and compensate for the reduced accuracy of directly optimizing B′ via deceptively optimizing B on the training data? I think this post provides us with tools to directly analyze this possibility. (If you buy the rest of the post, then with a sufficient speed + simplicity prior, the answer is that B will stay favored over B′. That's good!) Priors on Learned Optimizers Let's talk about priors! We can think of large neural networks as basically implementing short programs, and the process of "training" an NN is just searching through the space of programs until we find one that does well on our target task. We assume that if two NN-programs have equal performance, the training process will usually pick the one favored on priors/inductive biases. There are several different types of priors that neural networks might have: Simplicity prior/Solomonoff prior: if we converted the NN into a Turing Machine, how many states would writing down the TM require? In previous posts, Paul Christiano and others have argued that the Solomonoff prior is likely to favor deceptively-aligned mesaoptimizers (summarized by Mark Xu here). Speed prior: how many steps does the NN-turned-TM run for before terminating, on an average input? Evan Hubinger has argued whether the speed prior is heavily-disfavored in practice; he and I disagree about this, and I'l...