Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take: We're not going to reverse-engineer the AI., published by Charlie Steiner on December 1, 2022 on The AI Alignment Forum.As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes, which seems likely.Any approach to building safe transformative AI, or even just auditing possibly-safe TAI, which relies on reverse-engineering neural networks into fine-grained pseudocode based on mechanistic understanding should keep its ambitions very modest.This hot take is to some extent against ambitious "microscope AI," and to some extent against a more underlying set of intuitions about the form and purpose of interpretability research. (A somewhat related excellent background post is Neel's list of theories of impact for interpretability.) So I should start by explaining what those things are and why they might be appealing.Webster's Dictionary defines microscope AI as "training systems to do complex tasks, then interpreting how they do it and doing it ourselves." Prima facie, this would help with transformative AI. Suppose we're building some AI that's going to have a lot of power over the world, but we're not sure if it's trustworthy - what if some of its cognition is about how to do things we don't want it to be doing? If we can do microscope AI, we can understand how our first AI is so clever, and build a second AI that's just as clever and that we're sure isn't doing things it shouldn't, like running a search for how best to deceive us.Microscope-powered auditing is easier - if it's hard to assemble the second AI that does good things and not bad things, how about just checking that the first AI is trustworthy? To check an AI's trustworthiness in this microscope-AI-like framing of the issues, we might want to understand how its cognitive processes work in fine-grained detail, and check that none of those processes are doing bad stuff.When I say I'm against this, I don't mean auditing is impossible. I mean that it's not going to happen by having humans understand how the AI works in fine-grained detail.As an analogy, you can figure out how curve detectors work in InceptionV1. Not just in the sense that "oh yeah, that neuron is totally a curve detector," but in terms of how the whole thing works. It's yet more difficult to figure out that other neurons are not curve detectors - typically at this point we fall back on data-based methods like ablating those neurons and then trying to get the network to recognize rainbows, rather than first-principles arguments. But we can more or less figure out that InceptionV1 has an intermediate state where it detects curves, by an understandable algorithm and for human-understandable reasons.If we wanted to figure out how InceptionV1 tells dogs from cats, we might hope to gradually hack away at the edges - use what we know to expand the circle of knowledge a little more, and then repeat. Use the example of curve detectors to figure out spike detectors. Use spike-detectors to figure out fur-texture detectors, and curve detectors to figure out nose-shape detectors. Then we can learn how fur texture and nose shape play into deciding on dog vs. cat. At each step we can use data to test our understanding, but the basic goal is to be able to write down the flow of information between features in a human-comprehensible way. It's not just about giving neurons english-language labels, it's about giving them sensible algorithms where those labels play the expected role.The biggest problem with this plan is that neural networks leak. Many things are connected to many other things, weakly, in ways that are important for their success. I recently was at a talk that showed how the vast majority of attention heads in a transformer have lots of ...