Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How Do Selection Theorems Relate To Interpretability?, published by johnswentworth on June 9, 2022 on The AI Alignment Forum. One pretty major problem with today’s interpretability methods (e.g. work by Chris Olah & co) is that we have to redo a bunch of work whenever a new net comes out, and even more work when a new architecture or data modality comes along (like e.g. transformers and language models). This breaks one of the core load-pillars of scientific progress: scientific knowledge is cumulative (approximately, in the long run). Newton saw far by standing on the shoulders of giants, etc. We work out the laws of mechanics/geometry/electricity/chemistry/light/gravity/computation/etc, and then reuse those laws over and over again in millions of different contexts without having to reinvent the core principles every time. Economically, that’s why it makes sense to pump lots of resources into scientific research: it’s a capital investment. The knowledge gained will be used over and over again far into the future. But for today’s interpretability methods, that model fails. Some principles and tools do stick around, but most of the knowledge gained is reset when the next hot thing comes out. Economically, that means there isn’t much direct profit incentive to invest in interpretability; the upside of the research is mostly the hope that people will figure out a better approach. One way to frame the Selection Theorems agenda is as a strategy to make interpretability transferable and reusable, even across major shifts in architecture. Basic idea: a Selection Theorem tells us what structure is selected for in some broad class of environments, so operating such theorems “in reverse” should tell us what environmental features produce observed structure in a trained net. Then, we run the theorems “forward” to look for the analogous structure produced by the same environmental features in a different net. How would Selection Theorems allow reuse? An example story for how I expect this sort of thing might work, once all the supporting pieces are in place: While probing a neural net, we find a circuit implementing a certain filter. We’ve (in this hypothetical) previously shown that a broad class of neural nets learn natural abstractions, so we work backward through those theorems and find that the filter indeed corresponds to a natural abstraction in the dataset/environment. We can then work forward through the theorems to identify circuits corresponding to the same natural abstraction in other nets (potentially with different architectures) trained on similar data. In this example, the hypothetical theorem showing that a broad class of neural nets learn natural abstractions would be a Selection Theorem: it tells us about what sort of structure is selected for in general when training systems in some class of environments. More generally, the basic use-case of Selection-Theorem-based interpretability would be: Identify some useful interpretable structure in a trained neural net (similar to today’s work). Work backward through some Selection Theorems to find out what features of the environment/architecture selected for that structure. Work forward through the theorems to identify corresponding internal structures in other nets. Assuming that the interpretable structure shows up robustly, we should expect a fairly broad class of environments/architectures will produce similar structure, so we should be able to transfer structure to a fairly broad class of nets - even other architectures which embed the structure differently. (And if the interpretable structure doesn’t show up robustly, i.e. the structure is just an accident of this particular net, then it probably wasn’t going to be useful for future nets anyway.) Other Connections This picture connects closely to the converge...