Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AXRP Episode 17 - Training for Very High Reliability with Daniel Ziegler, published by DanielFilan on August 21, 2022 on The AI Alignment Forum. Google Podcasts link Sometimes, people talk about making AI systems safe by taking examples where they fail and training them to do well on those. But how can we actually do this well, especially when we can’t use a computer program to say what a ‘failure’ is? In this episode, I speak with Daniel Ziegler about his research group’s efforts to try doing this with present-day language models, and what they learned. Listeners beware: this episode contains a spoiler for the Animorphs franchise around minute 41 (in the ‘Fanfiction’ section of the transcript). Topics we discuss: Summary of the paper Alignment as scalable oversight and catastrophe minimization Novel contributions Evaluating adversarial robustness Adversary construction The task Fanfiction Estimators to reduce labelling burden Future work About Redwood Research In this transcript, to improve readability, first names are omitted from speaker tags. Filan: Hello everybody. Today, I’ll be speaking with Daniel Ziegler. After spending time as an ML engineer on the alignment team at OpenAI, Daniel is now the lead of the adversarial training team at Redwood Research. In this episode, we’ll be speaking about the paper ‘Adversarial Training for High-stakes Reliability’, on which he’s the first author. For links to what we’re discussing, you can check the description of this episode, and you can read the transcript at axrp.net. Welcome to the show, Daniel. Ziegler: Thanks. Summary of the paper Filan: So this paper: first of all, could you just summarize for us what’s in it, what it is, what it does? Ziegler: Sure. So basically, we took a pretty simple task that we think has some important analogous features to the kinds of AGI alignment situations we’re worried about in the future. And then we tried to attack it with adversarial training and see whether we could get really good adversarial robustness. So basically, we had a generator that was trained to generate fanfiction stories. It sees three sentences of an existing story, and it wants to generate one more sentence. And its task, or the task that we’re interested in, is to generate that sentence without introducing any new injuries that weren’t already present. And we’re using this as a stand-in for some catastrophic behavior that a future AI system could do. And so, our hope was: let’s take some generator that is pretty good at producing text, but maybe catastrophically fails in this way sometimes, and then use adversarial training to make it so that it never does that, no matter what inputs it’s given. Filan: And the way you’re doing this is by training some classifier and basically filtering the generator to not produce things that the classifier thinks introduces injuries? Ziegler: Exactly. So in this paper, we didn’t actually fine-tune the generator. We just had a classifier that filters its outputs. And then if you want to generate a safe output, you can keep drawing samples from the generator until you get one that the classifier’s happy with, and then you can be pretty sure that it’s good. Alignment as scalable oversight and catastrophe minimization Filan: And what do you see as the point of this paper? Do you see this as relevant for reducing x-risk from AI? And if so, how? Ziegler: So maybe I’ll zoom out a tiny bit and talk about how I think this fits into the overall AGI alignment picture. So I think one way to think about why AGI alignment is hard is a breakdown into two main pieces. This breakdown is basically due to Paul Christiano. He has a blog post called Low-stakes Alignment that goes into some of this. But I would say: the two main pieces are, one, that oversight is hard. So, your system is performing som...