Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Proxy misspecification and the capabilities vs. value learning race, published by Sam Marks on May 16, 2022 on The AI Alignment Forum. G Gordon Worley III recently complained about a lack of precision in discussions about whether Goodhart's Law will present a fatal problem for alignment in practice. After attending a talk in which Dylan Hadfield-Menell presented the "Goodhart's Law will be a big deal" perspective, I came away with a relatively concrete formulation of where I disagree. In this post I'll try to explain my model for this, expanding on my short comment here. More specifically, in this post I'll discuss proxy misspecification, which I view as a subproblem of Goodhart's Law (but not necessarily the entirety of it). Our model of proxy misspecification First I'll borrow Hadfield-Menell's mathematical model of the proxy misspecification problem. Suppose: Alice has things s1,.,sn which she values, i.e. she has a utility function U(s1,.,sn) which is increasing in each si. A robot is given a proxy utility function ~U which depends on some strict subset of the si's. The robot optimizes ~U by freely varying all si, subject only to resource constraints that require the si to trade off against each other. Then, as you'd expect, it's a theorem that the robot will set to their minimum possible values all si on which the proxy utility function ~U does not depend (instead putting all resources to increasing the si's which ~U does take into account). To borrow an example from the paper, we can imagine that the robot is a content recommendation algorithm. Then the si's might be metrics for things like ad revenue generated, engagement quality, content diversity, and overall community well-being. Humans care about all of these things, but our content recommendation algorithm might have a proxy reward that depends only on the first two. Furthermore, the algorithm only gets to recommend a limited amount of content, a resource constraint which (let's assume) prevents it from making all of these metrics as good as possible. In that case, according to the model, the algorithm will recommend content in a way that minimizes the features on which its proxy reward doesn't depend: content diversity and overall community well-being. My complaint: it might be hard to decouple the si's The above formulation of the proxy misspecification problem is a theorem. So if I'm going to resist the conclusion that future AI systems will destroy all value that their proxy rewards don't take into account, then I'll need to dispute part of the set-up or assumptions. Indeed, I'll attack the assumption that the robot is able to freely vary the si's subject only to resource constraints. In reality, many of the things we care about are highly correlated over "easily attainable" world states. Or in other words, for many pairs si,sj of things we care about, it's hard to increase si without also increasing sj. Furthermore, given a triple si,sj,sk of things we care about, it's even harder to increase si and sj without also increasing sk. And so on for quadruples, quintuples, etc. with it getting harder to decouple one thing we value from an n-tuple of other things we value as n grows larger. What work is being done by the word "hard" in the previous paragraph? One interpretation: finding ways to decouple the things human value and vary them freely requires sufficiently strong AI capabilities. And as the proxy utility function ~U takes into account more of the si and becomes a better approximation of U, the capabilities level needed to decouple the si's on which ~U depends from the rest increases. And if you buy all of this, then a corollary is: for each fixed capabilities level, there is an acceptable "margin of error" for our proxy utility functions such that if ~U is within this margin of error from U, the...