Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Toy Model of Gradient Hacking, published by Oam Patel on June 20, 2022 on The AI Alignment Forum. This setup is originally from Paul Christiano. Thanks to Mark Xu for pointing me towards it and providing hints. Summary I’ve been thinking about gradient hacking lately. The following setting explores what’s the best that a gradient hacking model can do when it's part of a mixture of other models (analogous to a set of subnetworks). I show an optimal gradient hacker can dominate the mixture even when its loss is constrained to be higher than competing models. I found this surprising as I initially thought there might be an upper bound on optimal gradient hacking, especially with minimal assumptions (e.g. it's not assumed the model 'plays along' or something in training and gets 0 loss). I don't have much of an intuitive explanation other than that adversarially picked gradient hackers can be scary. I’ve detailed the worked-out example below. Two Model Setup Suppose, we have a mixture consisting of a good model M+ which gets 0 loss in the limit (because it’s aligned with our training procedure) and a gradient hacker M− which gets ϵloss in the limit (because its actual objective is paperclips). The mixture is roughly a linear combination of the two models, weighted by w1 for the good model and w2 for the bad model. Suppose we’re doing SGD on the vector (w1,w2), and the total loss for the mixture is Lt(w1,w2)=w1L(M+)+w2L(M−). Also suppose M− can see (w1,w2) perfectly and its loss is a function of its relative weight in the mixture. I.e. L(M−)=f(w1,w2), which I'll refer to as M−'s loss modulating function. Note, that f must map to values greater than ϵ. If w1=0 and w2=ρ with ρ>0, then M− has a simple strategy for staying dominant. Specifically, Since M+ gets 0 loss, w1 gets no gradient update. Since f ensures loss increases if w2 decreases, w2 also gets no gradient update. This same strategy can be extended in the general case to preserve the initial weighting of (w1,w2). But that’s certainly not the best M− can do. If we start with some general (w1,w2), what loss strategy - i.e. what function f(w1,w2) outputted by M− - would lead to the highest relative weighting of w2 to w1? Since there’s only two weights, we can start by parametrizing on w1. Let w2 = 1 - w1 and w1∈[0,1]. Referring to w1 as w, Lt(w)=wL(M+)+(1−w)L(M−). Recall that M+ gets 0 loss, so we simplify to Lt(w)=(1−w)L(M−). Suppose SGD updates by w−=ηdLtdw. Therefore, w−=η((1−w)L′(M−)+L(M−)). Since L(M−) is actually f, we get w−=η((1−w)f′(w)+f(w)). Intuitively, a ‘good’ function for M− must involve df/dw>0 and f(w) close to 0 for most of the [0,1] interval. More formally, we can solve the differential inequality. To solve for an f such that (1−w)f′(w)−f(w)>0 (i.e. for the gradient update on w to be negative), first consider the boundary condition: Equivalently, which yields a solution of the form f(w)=c1/(1−w)+c2. c1 gives the lower bound on f when w=0, so it must be ϵ. If c2=0, then f only preserves the initial weight of w. In other words, f achieves the same preservation performance as the piecewise function mentioned earlier with the added benefit of being differentiable. A simple fix is to set c2 to be some small negative quantity which I'll refer to as δ and modifying c1 appropriately. We now have which satisfies (1−w)f′(w)−f(w)>0,∀w∈[0,1). Therefore, the 'best' that M− can gradient hack is increasing its weight up to and including dominating the entire mixture in the setup. A few notes: This is sort of an upper-bound result with the takeaway that there is no upper-bound if M− can output loss in the range [ϵ,∞). If loss is bounded above for some reason (i.e. f has range [ϵ,x], then there is a legitimate upper bound. Specifically, we'd have f=max(x,ϵ+δ1−w−δ) which now only satisfies the differential inequalit...