Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An observation about Hubinger et al.'s framework for learned optimization, published by Spencer Becker-Kahn on May 13, 2022 on The AI Alignment Forum. The observations I make here have little consequence from the point of view of solving the alignment problem. If anything, they merely highlight the essential nature of the inner alignment problem. I will reject the idea that robust alignment, in the sense described in Risks From Learned Optimization, is possible at all. And I therefore also reject the related idea of 'internalization of the base objective', i.e. I do not think it is possible for a mesa-objective to "agree" with a base-objective or for a mesa-objective function to be “adjusted towards the base objective function to the point where it is robustly aligned.” I claim that whenever a learned algorithm is performing optimization, one needs to accept that an objective which one did not explicitly design is being pursued. At present, I refrain from attempting to propose my own adjustments to the framework, or to build on the existing literature or to develop my own theory. I am certainly not against doing any of those things, but they are things to possibly be pursued later; none of them is the purpose of this post. To make my main point, I will introduce only a bare minimum of mathematical notation. We will show that a mesa-objective always has a different type signature to a base objective and that the default assumption ought to be that there is no way to compare them in general and certainly no general way to interpret what it means for them to ‘agree’. Suppose that an optimizer is searching through a space S of systems. At this time, I do not want to attempt to unpack what it means to 'search', but, naively, we can imagine that there is an objective function f :S→R, which determines something that we might call the 'search criterion'. The idea of course is that the optimizer is a system that is 'searching' through the set S and judging different points according to the criterion that higher values of f are better. In the background, there is some 'task' and naively we can think of this as being represented by a 'task space' X which consists of all of the different possible 'presentations' or 'instances' of the task. For example, perhaps the task is choosing the next move in a game of Go or the next action in a real-time strategy video game. In these examples, a given x∈X would represent a board position in Go, say, or a single snapshot of the game-state in the video game. Then, in general, given x∈X and s∈S, we can think that s(x) is the output of s on the task instance x or the action taken by s when presented with x (i.e. s(x) denotes the next board move in Go or the next action to be taken in the video game). So each element of S defines a map from the task space X to some kind of output space or space of possible actions, which we need not notate. Now, it is possible that there exists m∈S which works in the following way: Whenever the output of m on an instance x of the task needs to be evaluated, i.e. whenever m(x) is computed, what happens is that m searches over another search space Σ and looks for elements that score highly according to some other objective function g:Σ→R. Whenever this is the case, we say that such an m∈S is a mesa-optimizer and that the original optimizer - the one that searches over S - is the base optimizer. Notice that in some way, elements of Σ must in turn correspond to outputs/actions, because given some x, the mesa-optimizer m conducts a search over Σ to determine what output m(x) is, but that is all just part of the internal workings of m and we need not 'know' or notate how this correspondence works. In Risks From Learned Optimization, Hubinger et al. write: In such a case, we will use base objective to refer to whateve...