Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reframing inner alignment, published by davidad (David A. Dalrymple) on December 11, 2022 on The AI Alignment Forum.The standard frame (Evan Hubinger, 2021) is:Outer alignment refers to the problem of finding a loss/reward function such that the training goal of “a model that optimizes for that loss/reward function” would be desirable.Inner alignment refers to the problem of constructing a training rationale that results in a model that optimizes for the loss/reward function it was trained on.Here’s the reframe (I believe the credit for this breakdown is due to John Wentworth, although I haven't found anything online to link to for it):Reward Specification: Finding a policy-scoring function J(π) such that (nearly–)optimal policies for that scoring function are desirable."Are you optimising for the right thing?"Adequate Policy Learning: Finding a policy that’s actually (nearly–)optimal for that scoring function."Did you optimise for it enough?"As Paul Christiano points out (in an excerpt recently highlighted by Alex Turner to make a similar point), factoring out Reward Specification represents only one "particular kind of alignment strategy, that's like a two step plan" for how one might try to conclude that a learned policy is desirable, where we first align a scoring function J with what we actually want, and then align a training process with our scoring function. Under this kind of plan, the proof tree for overall existential safety would conclude by applying an implication like this:Mesa-optimisersAt this point you might be wondering: what happened to the concept that π might contain a mesa-optimiser for something different from J?The proper role of the mesa-optimiser concept is in an explanation of why Adequate Policy Learning is insidiously difficult:In machine learning, one typically approximates (the gradient of) the reward function by aggregating a set of evaluations on only a finite sample (the empirical “training distribution”), whereas the true J also depends on what π does globally, even in rare states that may never be sampled during training.If the trajectories observed in deployment have a state distribution which diverges from the training distribution, then the policy that was optimal on the training set will likely diverge from the optimal policy. This is standard out-of-distribution generalisation failure.If the policy that was optimal on the training set includes a good understanding of the real world and consequentialist reasoning (which is likely, if the training set is sufficiently rich that you can get a policy that's transformative in the real world), then it contains a capable mesa-optimiser, and it's likely that it's still transformative in the real world, even while experiencing generalisation failure: capabilities generalise better than alignment.The bottom line is, once we get beyond some level of capabilities, it becomes likely that the gap between the empirical training distribution and the intended reward function suddenly gets exploited in a pretty worst-case-like way (i.e. optimising for some arbitrary thing, instead of merely being bad at optimising the outer objective). I think this phenomenon is also what the concept of a sharp left turn refers to, so this post could equally also be titled "Reframing the sharp left turn".Inner alignmentThe standard "inner alignment" answer to "what do we do about this?" is, to quote Paul again,let’s build a system that’s internalized that objective in some sense, or is not doing any other optimization beyond pursuit of that objective.In other words: let's make sure that the policy π contains a mesa-optimiser for the outer objective J, or at least, contains no other mesa-optimisers.If we assume outer alignment and Reward Specification are solved, the above notion of inner ali...