Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A naive alignment strategy and optimism about generalization, published by Paul Christiano on the AI Alignment Forum. (Context: my last post was trying to patch a certain naive strategy for AI alignment, but I didn’t articulate clearly what the naive strategy is. I think it’s worth explaining the naive strategy in its own post, even though it’s not a novel idea.) Suppose that I jointly train an AI to do some task (e.g. make money for me) and to answer a wide range of questions about what is happening in the world (e.g. “why did Alice just wire $1000 into my bank account?” or “what is Bob thinking right now?”). I generate training data for the QA task in a really simple way: I choose a subset of questions that humans are able to reliably answer, and use those as a training set for supervised learning. I’ll call this the naive training strategy. I’d like for my AI to tell me everything it knows. If the AI bought a stock because it expects a merger announcement soon, I want it to tell me about the predicted merger announcement. If the AI predicts a merger announcement because it inferred that executives of the companies have been in extensive talks over the last month, I want it to tell me about those talks. I’m not asking the AI to explain why it made a given decision, I’m asking the AI to tell me as much as it can about the world. The important property is that if the AI “knows” something and uses that knowledge to perform the task well, then it also uses that knowledge to answer questions well. Why might this work? The hope is that “answer questions honestly to the best of your ability” is a natural thing for our AI to learn — that there is some simple way to translate from the AI’s model of the world into natural language and to honestly report what it believes. If our training dataset is good, then this policy will score well, and we can hope that SGD will find it. I’ll call this the intended policy. Why might this not work? The concern is that “predict how a human would answer questions” is also a natural thing for our AI to learn, especially if the AI is doing a task that already requires predicting humans. Predicting humans also gets a low loss on the training set, but it generalizes poorly once we start asking our AI questions that a human couldn’t have answered on their own. If we’re worried about this we could use a different loss function than “predict what humans would say.” But regardless of what loss function we choose, the policy could still be trying to game the loss function. If SGD learns to “game the loss function” then we’ll generalize poorly in any case where humans can’t distinguish a good question-answerer from a bad question-answerer. I’ll call “game the loss function” the instrumental policy. It answers questions well only because doing so is instrumentally useful for getting a low loss, and so it will start answering questions badly if that situation changes. (This is closely related to deceptive alignment, but I think the concept makes sense without defining alignment or optimizers; there’s also a difference in emphasis in that I’m often considering models which are “intrinsically motivated” to game the loss function rather than doing so in order to gain influence. I first introduced this term in Inaccessible Information.) What I’m doing now I’m trying to dig into a bunch of reasons why the naive training strategy might fail, and to understand whether there is a way to modify the naive strategy to avoid those problems. In my last post I discussed two reasons that the naive training strategy might learn the instrumental policy instead of the intended policy: There may be predictable errors in the training data, such that instrumental policy actually gets a lower loss than answering honestly (because it responds strategically to errors). If the...