Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reverse-engineering using interpretability, published by Beth Barnes on December 29, 2021 on The AI Alignment Forum. Building a model for which you're confident your interpretability is correct, by reverse-engineering each part of the model to work how your interpretability says it should work. (Based on discussion in alignment reading group, ideas from William, Adam, dmz, Evan, Leo, maybe others) The basic idea Train an untrusted network Do interpretability on it, getting some human-understandable schema that is intended to capture what (some of) the model is doing Identify anything that should be changed about what the model is (supposedly) doing, and make a new human-understandable schema of the desired behavior Train a model to do what this new representation says it should be doing, by having a human implement each part of the schema then doing imitation + distillation I think this can be thought of in a few different ways: One way you might implement Microscope AI in practice A way to do imitative amplification where the decomposition is generated by gradient descent instead of by humans, combined with some aspects of imitative generalization The main advantages here are: Compared to just using interpretability to audit your trained model, this is safer, because you can be more confident your interpretation matches what the final model is doing (because you trained the final model to conform to your interpretation). If you have mistakes in your interpretability, you should get performance issues rather than safety issues. Compared to normal IDA or ‘trying to build a schema from scratch’, this is easier because you use the power of the initial model to decide how to decompose tasks, or how to solve individual tasks, and you can use patterns the model learnt from large volumes of data without needing to have been able to generate that knowledge yourself The main disadvantages are: To be competitive this requires really good interpretability - basically understanding everything about what the model is doing. This scheme might actually require more out of your interpretability techniques than just auditing your model would. The ‘ontology mismatch’ issue for ELK is still a problem. A naive implementation of this training procedure is really expensive (in terms of both compute and human labels) The scheme in more detail: First, train some big, untrusted model on untrusted, unsupervised data Next, use some (automated) interpretability tools to get natural-language (or otherwise) descriptions of functions of components of the model (e.g. neurons, circuits, attention heads, etc) E.g.: Now we’re going to build a new model that is constructed based on the description of this model. Each component in the new model is going to be a small model trained to imitate a human computing the function that the description of the component specifies. For the low-level components, we train a small model to e.g. recognise edges, based on human labels of whether a region contains edges. For the components above this, we train a small model to go from the human-understandable descriptions of the features that go into it, to the human-understandable description of behavior. For example, we train a model to go from the description of the locations of windows, wheels and car body in the image to a description of the location of likely cars in the image. Now we have a model that is definitely well-described by the interpretability we did, because we constructed each component to have the function we understood it to. (There’s another assumption needed here - that we can ‘trust’ IID supervised learning. That is, if we have a dataset X, and we can get human labels h(x) for any particular example x \in X, we can train a model to perfectly imitate h on X.) The hope is that we can now do things l...