Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concrete Steps to Get Started in Transformer Mechanistic Interpretability, published by Neel Nanda on December 25, 2022 on The AI Alignment Forum.IntroductionFeel free to just skip the intro and read the concrete stepsThe point of this post is to give concrete steps for how to get a decent level of baseline knowledge for transformer mechanistic interpretability (MI). This is an experiment in trying to give concrete, actionable, goal-oriented advice that I think is enough to get decent outcomes. Naturally, this can be constraining and the best path will differ a lot between your background and precise goals! As an accompaniment, I’m writing a future post fleshing out my philosophy behind doing mechanistic interpretability research, what I think the sub-skills are, and fleshing out how I think people should learn them.A core belief I have about learning mechanistic interpretability is that you should spend at least a third of your time writing code and playing around with model internals, not just reading papers. MI has great feedback loops, and a large component of the skillset is the practical, empirical skill of being able to write and run experiments easily. Unlike normal machine learning, once you have the basics of MI down, you should be able to run simple experiments on small models within minutes, not hours or days. Playing around with models builds this empirical skill, but also helps enhance the reading and learning. It builds intuitions for how the model actually works internally, what techniques are easy vs hard, etc, which is key context when understanding the ideas in papers.Further, because the feedback loops are so tight, I don’t think there’s a sharp boundary between reading and doing research. If you want to deeply engage with a paper, you should be playing with the model studied, and testing the paper’s basic claims. And if you’re doing this, you can try to answer the questions that come up as you read the paper. And there’s a smooth continuum between this and doing real original research.The intended audience is for people new-ish to Mechanistic Interpretability but who know they want to learn about it - if you have no idea what MI is, check out my MI Explainer or Circuits: Zoom In.Defining “Decent Baseline”Scoping out my goals here, I want you to be able to take a behaviour in a transformer that you want to understand, and have some idea of how to get started and get traction. Breaking this down further:A good grounding in the key concepts of ML and MIAn intuition for how a transformer actually works as a mathematical object - what the moving parts are, how it all fits together, and how to reason about the overall systemFamiliarity with tooling, such that you can easily spin up a model and run quick and dirty experiments.A rough map of the literature, what’s known in the field, and big categories of open problems - not necessarily a deep knowledge, but hopefully enough to get a sense for techniques used, and where you could go and read a relevant paper if you want to.A sense of basic techniques, what compelling evidence about model internals looks like, and how to get started when poking around at a model.This is a deliberately limited framing! Importantly, this does not mean having a deep knowledge and understanding of everything that’s known in the field, nor the skills to actually produce important novel research - these are much harder to gain, especially the second! I consider these out of scope for this post.Further, I focus on the specific skill of getting started at reverse engineering a system because it’s much more concrete than other skills, yet also fundamental and important. Not all MI research explicitly looks like “take a system and try to reverse engineer it”, and often significant skill goes into identifying which sy...