Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transparency Trichotomy, published by Mark Xu on the AI Alignment Forum. Introduction In , Evan presents a trichotomy of ways to understand a model M: Transparency via inspection: use transparency tools to understand M via inspecting the trained model. Transparency via training: incentivize M to be as transparent as possible as part of the training process. Transparency via architecture: structure M's architecture such that it is inherently more transparent. For each type of transparency, we will analyze them on dimensions that influence how effectively we can use them to align powerful systems: Scalability asks how easily the transparency method scales to more parameters, compute, data, training steps, etc. Factors that influence scalability include reliance on human judgment, the ability to automate with compute, and how easily the human judgment required can be distributed. Gameability asks the extent to which optimizing an interpretability tool makes it less valuable. Easily gameable tools can be rendered useless by SGD or mesa-optimizers. Factors that influence gameability include how frequently the transparency tool is deployed, how much the tool relies on human judgment, and how it is used. Quality asks about the level and type of understanding the transparency tool can give you about how the model works. Factors that influence quality include reliance on human judgment, the smallest unit at which the tool can operate, and how sharp the resulting predictions are. Transparency via inspection Transparency via inspection seeks to understand and predict a model's behavior via looking at it directly. Currently, this means looking at the weights of a neural network and trying to obtain information. A prototypical example of inspection transparency is Olah et al.'s Circuits thread. Scalability. Inspection transparency primarily functions by having humans look at the weights of a neural network and make various judgment calls. For instance, the authors of An Overview of Early Vision in InceptionV1 look at the first layer of InceptionV1 and determine that 28 of the 64 features correspond to Gabor filters. As such, current inspection transparency requires large amounts of human labor, making it scale poorly with parameter count (unless labor can be automated). Current inspection transparency tools often use dataset examples to determine the function of particular neurons, so these tools might scale with increased data. It's possible that once neural networks are better understood, inspection transparency can be mostly automated, with human judgment only required for edge cases. Additionally, Filan et al.'s Pruned Neural Networks are Surprisingly Modular suggests that inspection transparency might scale softer than expected. Gameability. Since inspection transparency requires many human judgment calls, it is approximately as gameable as human judgment. To this extent, inspection transparency tools can be gamed insofar as there is something applying optimization power to game them. We postulate that the most likely source of this optimization power is a mesa-optimizer. In particular, inspection transparency tools that easily scale are more likely to be proxy measures, making them more gameable. To decrease the chances of getting gamed, one can try to sanitize the training data to remove information that would make such gaming possible. The extreme end of sanitization is Garrabrant and Kumar's suggestion that human models be avoided, which can concretely be implemented in . If one is less concerned about deceptive alignment, inspection transparency tools might provide an independent check against by, e.g., verifying that a classifier relies on features that a human would consider robust. Quality. The amount of information we could gain about a neural network by looking at ...