Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introduction to inaccessible information, published by Ryan Kidd on December 9, 2021 on The AI Alignment Forum. This post was written under Evan Hubinger's direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program. TL;DR: If we want to understand how AI models make decisions, and thus assess alignment, we generally want to extract their internal latent information. That information might be inaccessible to us by default and defy easy extraction. Resolving this problem might be critical to AI alignment. Paul Christiano's conception of 'inaccessible information' seems to be an important and useful lens for interpreting many key concepts in AI alignment. In this post, I will attempt to explain (in an accessible way): What inaccessible information is; How inaccessible information might arise in ML models; and What insight inaccessible information offers for alignment research. What is inaccessible information? Roughly, inaccessible information is anything a model knows that an external agent cannot reliably learn. For instance, in the process of training, a sufficiently large model might acquire a complicated internal structure that is opaque to direct inspection. If we want to understand the process by which this 'black box' turns inputs into outputs, perhaps to learn new theories/heuristics, check for undesirable future behaviour or inspect the model's limitations (e.g. myopia), it seems natural to also train the model to produce explanations for its outputs. However, the model might learn to produce convincing explanations that are not necessarily true representations of its input-output process. If true and false but convincing explanations are indistinguishable to the training mechanism (e.g. a human with a reward button), the model may converge to the instrumental policy: 'give whatever explanation is most convincing'. This scenario is clearly counter to the desired outcome, as the model does not learn to give strictly true explanations and therefore its outputs are contingent on inaccessible information. Generally, we might be unable to distinguish true and false but convincing explanations of a model's output if: We cannot directly check that an explanation is true (e.g. 'Alice refused icecream because she was depressed'); and We cannot build a trusted model that enables us to check the original model (e.g. a trusted short-term weather forecaster that we can extrapolate to test a long-term hurricane forecaster). A related property to 'explainability' is 'transparency', also called 'interpretability'. A model is transparent to an external agent if inspection of the model is sufficient to understand how it operates. From the perspective of such an external agent, more transparent models generally contain less inaccessible information. Model transparency can be increased by: Improving model inspection tools; Training a model to be more accessible; and Structuring a model's architecture to be inherently accessible. How might inaccessible information arise in ML models? The training-induced structure of a machine learning model can be thought of as containing 'latent information' that shapes the model's output. Models that perform well on complicated tasks in diverse environments (e.g. 'What will Alice say next?') might acquire latent information that aids predictions (e.g. a model of Alice's thoughts). Larger models, or models that include highly compressed representations of the environment (i.e. big world, small map), are generally less accessible to direct inspection. A model's latent information might be difficult to render accessible by training it to give explanations or be otherwise transparent if, for example, there is: Training incentives that favour the instrumental policy; Learned dece...