Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is:Outer alignment and imitative amplification, published by Evan Hubinger on the AI Alignment Forum. Understanding the outer alignment problem What really is outer alignment? In “Risks from Learned Optimization,” we defined outer alignment in the context of machine learning as “aligning the specified loss function with the intended goal.” But that's not a perfectly well-defined statement—what does it mean for a loss function to be “aligned” with the intended goal? If the goal we care about is maximizing U , do we need exactly L − a U b for constants a b ? That's a pretty high bar. Well, what exactly do we want outer alignment for? At the end of the day, we care about whether the model that pops out the other end of our training procedure will be safe, which is a complicated question involving the loss function, the architecture, the implicit inductive biases, and so on. In what sense, then, is it even reasonable to look at just the lost function in isolation and ask whether it's aligned or not? I think the strongest case for outer alignment being a meaningful problem in isolation comes from the argument that loss functions seem to scale pretty well with generic machine learning progress. If, as a silly example, your outer alignment scheme is to “train image classification models,” that's something that ML has progressively gotten better at over time. Compare that to the silly inner alignment scheme of “train a straightforward CNN”—that's something that ML has passed by pretty rapidly in favor of architectural improvements like residual connections even just for the task of image classification. Of course, outer alignment alone does not an aligned AGI make, so you still have to have some notion of how you're going to do inner alignment in mind—but loss functions scaling better is still a perfectly valid reason for focusing on outer alignment.[1] Thus, it does seem quite reasonable to me to put effort into finding “aligned” loss functions. But that still brings us back to the question of what exactly makes a loss function “aligned.” In the context of a specific training/inner alignment scheme, we can say that a loss function is aligned if, when plugged into that training scheme, it produces models which are aligned with our goals. But in the absence of any specific training scheme, what does it mean to say that a loss function is aligned in isolation? We can of course ask for L − a U b as I stated previously, though in my opinion I think achieving something like that is likely to be nearly impossible. Outer alignment at optimum I think there is another version of “outer aligned in isolation,” however, which is both meaningful and (at least somewhat) achievable which I will call outer aligned at optimum. Intuitively, I will say that a loss function is outer aligned at optimum if all the possible models that perform optimally according to that loss function are aligned with our goals—that is, they are at least trying to do what we want. More precisely, let M X → A and L X → A → R M → R . For a given loss function L ∈ L , let l ∗ min M ∈ M L M . Then, L is outer aligned at optimum if, for all M ∗ ∈ M such that L M ∗ l ∗ M ∗ is trying to do what we want. That's the definition—now why should we care? In basically any practical setting we're never going to reach perfect loss, so why should it matter if those functions which do have perfect loss are aligned or not? In my opinion, I think there is a strong argument for loss functions which are aligned at optimum being significantly less susceptible to Goodhart's Law as we scale up ML capabilities. Suppose you know that a loss function L is aligned for current ML capabilities. When you then scale up those capabilities and push harder on minimizing L , you immediately run into all the issues of Goodhart's Law where L can quickly ce...