Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Outer vs inner misalignment: three framings, published by Richard Ngo on July 6, 2022 on The AI Alignment Forum. A core concept in the field of AI alignment is a distinction between two types of misalignment: outer misalignment and inner misalignment. Roughly speaking, the outer alignment problem is the problem of specifying an reward function which captures human preferences; and the inner alignment problem is the problem of ensuring that a policy trained on that reward function actually tries to act in accordance with human preferences. (In other words, it’s the distinction between aligning the “outer” training signal versus aligning the “inner” policy.) However, the distinction can be difficult to pin down precisely. In this post I’ll give three and a half definitions, which each come progressively closer to capturing my current conception of it. I think Framing 1 is a solid starting point; Framings 1.5 and 2 seem like useful refinements, although less concrete; and Framing 3 is fairly speculative. For those who don’t already have a solid grasp on the inner-outer misalignment distinction, I recommend only reading Framings 1, 1.5 and 2. For the purposes of this document, I’ll focus on the reinforcement learning setting, since that’s the case where the distinction is clearest. However, the same concepts could also apply in a supervised or self-supervised context, if we replace references to “policies” and “reward functions” with “models” and “loss functions”. The extent to which the inner alignment problem will appear in non-RL contexts is an important open question in alignment. Framing 1: types of behavioral misalignment Consider training an RL policy until it’s getting high reward in its training environment. Suppose we then evaluate the policy in a different test environment, without retraining it; what will happen? Consider four possibilities: The policy behaves incompetently. This is a capability generalization failure. The policy behaves in a competent and desirable way. This is aligned behavior. The policy behaves in a competent yet undesirable way which gets high reward according to the original reward function. This is an outer alignment failure, also known as reward misspecification. The policy behaves in a competent yet undesirable way which gets low reward according to the original reward function. This is an inner alignment failure, also known as goal misgeneralization. Langosco et al. (2022) provide a more formal definition and some examples of goal misgeneralization. Note that this categorization requires us to evaluate the original reward function in the new environment. This makes sense when the environments are fairly similar (e.g. “playing Go against an AI” vs “playing Go against a human”) or the reward function is fairly general (e.g. based on human evaluations). However, we should be most concerned about alignment failures in policies that are able to behave competently in test environments very different from their training environment (e.g. generalizing from simulations to the real world, or from one task to another). Under such large shifts, there may be many reasonable ways to generalize the original reward function to the test environment (even for human evaluators), which blurs the distinction between reward misspecification and goal misgeneralization. The next framing aims to provide a definition of alignment which avoids this problem. Framing 1.5: causes of behavioral misalignment When a policy misbehaves in a test environment, the thing we intuitively care about is what caused that misbehavior. Since we’re not doing any additional training in the test environment, this doesn’t depend on how we evaluate the original reward function in the test environment. Instead, it depends on the rewards the policy received in the training environmen...