Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Comparing Four Approaches to Inner Alignment, published by Lucas Teixeira on July 29, 2022 on The AI Alignment Forum. Early work on this was supported by CEEALAR and was finished during an internship at Conjecture under the mentorship of Adam Shimi. Thank you to Evan Hubinger and Rohin Shah for answering some questions related to this post. Epistemic Status: First palimpsest of many to come. Intro and Motivation I am broadly interested in pluralism in scientific development, and more practically interested in supporting the diversification of conceptual alignment. Towards that end, I believe that having a systematic analysis of the cruxes and methodologies which split up the field into distinct agendas could help clarify how exactly diversity is created and sustained in our field, and what exactly it is we wish to diversify in order to better manage our collective portfolio of research bets. As a case study, this post will investigate four different approaches to inner alignment. I’ll be taking a look at the different definitions which they use for “outer alignment” and conjecturing on how: Despite inconsistencies across approaches, the different definitions utilized by each approach maintain coherent when understood against the backdrop of the aims local to that approach Which cruxes set these different approaches apart. This post is distillational in nature, and as such most of the ideas which I present here are not novel and not my own. The claims in this post should also be read as a part of an ongoing exploratory process, and any pushback (especially from those whose work I cite) would be beneficial. The Approaches The Mechanistic Approach The mechanistic approach deconfused a lot of previous work on optimization daemons and provided the now canonical argument for why we’d expect mesa-optimizers and inner misalignment, which is summarized as follows: Traditionally, most of the optimization pressure (or in other words, searching procedures) is implemented looking for a specific model which performs competently on a task during training time. However, as ML becomes more and more powerful, models will be trained to more complicated tasks across a wider range of environments. This puts more pressure for a model to generalize better, and raises the incentives for the model to delegate some of its optimization power to deployment time, rather than training time. It is in those cases where we get a mesa-optimizer, and it is with emergence of mesa-optimizers where the issues of misalignment between the mesa-optimizer’s objectives and the objectives which we were training for arise. The mechanistic approach has also created different evaluations and methods of evaluations of alignment proposals. This approach is defined by an aim towards conceptual clarity, and not only in the employment of mechanistic definitions in both proposed solutions and problem statements, but also with their continued refinement. The Empiricist approach In contrast to the mechanistic approach, the empirical approach’s strategy is mostly focused on developing knowledge of inner alignment by creating empirical experiments studying the phenomena. Tightly coupled to this strategy is a favoring of empirical operationalizability over mechanisticality as a precondition for the definitions which they use. They will often allude to the intentional stance as an influence and cite 2-D Robustness as a core frame for the alignment problem. They favor terms such as objective robustness and generalization over mesa-optimizers and inner alignment. A redrafting of the classical argument for why we’d see inner misalignment in their language is as follows: Under certain non-diversified environments, a set of actions may be coherent with the pursuit of more than one goal, call them G_1 and G_2, where G_2 will refer...