Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Operationalizing compatibility with strategy-stealing , published by Evan Hubinger on the AI Alignment Forum. Thanks to Noa Nabeshima and Kate Woolverton for helpful comments and feedback. Defining optimization power One of Eliezer's old posts which I think has stood the test of time the best is his “Measuring Optimization Power.” In it, Eliezer defines optimization power as follows.[1] Let A be some action space and p be some probability measure over actions. Then, for some utility function U A → R and particular action a ∗ ∈ A , Eliezer defines the bits of optimization power in a ∗ as opt p U a ∗ − log 2 ∫ a ∈ A U a ≥ U a ∗ p a d a which, intuitively, is the number of times that you have to cut the space in half before you get an action as good according to U as a ∗ In my opinion, however, a better, more intuitive version of the above definition can be obtained by using quantilizers. A q -quantilizer relative to some utility function U and base distribution over actions p is a system which randomly selects an action from the top q fraction of actions from p sorted by U . Thus, a 0.1 -quantilizer selects actions randomly from the top 10% of actions according to U . Intuitively, you can think about this procedure as being basically equivalent to randomly sampling 1 q actions from p and picking the best according to U Now, using quantilizers, we can give a nice definition of optimization power for an entire model. That is, given a model M , let q ∗ ∈ 0 1 be the smallest fraction[2] such that a q ∗ -quantilizer with base distribution p is at least as good[3] at satisfying U as M . Then, let opt p U M − log 2 q ∗ . What's nice about this is that it gives us a measure of optimization power for a whole model and a nice intuitive picture of what it would look like for a model to have that much optimization power—it would look like a q ∗ -quantilizer. Both of these definitions do still leave the distribution p unspecified, but if we want a very general notion of optimization power then I would say that p should probably be some sort of universal prior such that simple policies are weighted more heavily than their more complex counterparts. If we use the universal prior, we get the nice property that the more complex the policy needed to optimize some utility function, the more optimization power is needed. Thus, we can replace opt p U with just opt U where p is assumed to be some universal prior. Compatibility with strategy-stealing Now, given such a definition of optimization power, I think we can give a nice definition of what it would mean for an AI system/training procedure to be compatible with the strategy-stealing assumption. Intuitively, we will say that an AI system/training procedure train U → M which maps utility functions onto models is compatible with strategy-stealing if opt U train U doesn't vary much over some set of utility functions Y that is, if train isn't better at optimizing for (or producing models which optimize for) some objectives in Y than others. We can make this definition more precise for a set of utility functions Y if we ask for stdev opt U train U U ∈ Y to be small.[4] This definition is very similar to my definition of value-neutrality, as they are both essentially pointing at the same concept. What's nice about using opt U here, though, is that it lets us compare very difficult-to-satisfy utility functions with much easier-to-satisfy ones on equal footing, as we're just asking for train to produce actions which always score in the top whatever percent—which should be equally easy to achieve regardless of how inherently difficult U is to satisfy.[5] Notably, this definition of compatibility with strategy-stealing is somewhat different than others' notions in that it is about a property of a single AI system/training procedure rather than a ...