Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AXRP Episode 16 - Preparing for Debate AI with Geoffrey Irving, published by DanielFilan on July 1, 2022 on The AI Alignment Forum. Google Podcasts link Many people in the AI alignment space have heard of AI safety via debate - check out episode 6 of AXRP if you need a primer. But how do we get language models to the stage where they can usefully implement debate? In this episode, I talk to Geoffrey Irving about the role of language models in AI safety, as well as three projects he’s done that get us closer to making debate happen: using language models to find flaws in themselves, getting language models to back up claims they make with citations, and figuring out how uncertain language models should be about the quality of various answers. Topics we discuss: Status update on AI safety via debate Language models and AI safety Red teaming language models with language models GopherCite Uncertainty estimation for language reward models Following Geoffrey’s work, and working with him Daniel Filan: Hello, everybody. Today I’ll be speaking with Geoffrey Irving. Geoffrey is a safety researcher at DeepMind who leads the scalable alignment team. We’ll be speaking about three papers he’s co-authored: Red Teaming Language Models with Language Models, whose first author is Ethan Perez; Teaching Language Models to Support Answers with Verified Quotes, aka the GopherCite paper, whose first authors are Jacob Menick, Maja Trebacz and Vladimir Mikulik; and Uncertainty Estimation for Language Reward Models, whose first author is Adam Gleave. For links to what we’re discussing, you can check the description of this episode and you can read the transcript at axrp.net. Welcome to the show, Geoffrey. Geoffrey Irving: Thank you. Status update on AI safety via debate Daniel Filan: So I guess my first question is, what happened to AI Safety via Debate? Geoffrey Irving: It is the most important question. So I think the thing that happened is I’m still doing that stuff. I’m building up towards it. And the overall research agenda here is figure out and implement the protocol for humans and machines discussing problems that gives us good answers that we endorse after reflection. So that’s debate, but then broadened: there’s a lot of different versions of debate. There’s different things you can add onto it. You could do debate plus evidence or debate plus more aggressive red teaming. There’s a large space of protocols there. Geoffrey Irving: And these papers are kind of pieces of that story, but we were also kind of working on just pushing on the main thing. I think that will take some time to put together because there’s still pieces to build up. But that is still very much the agenda, or a piece of the larger agenda. An example of a thing which I’ve updated on since then is the human needs to talk. The debate paper only had two machines talking, and then at the end a human judges. And that’s just not the right thing to do, obviously, because humans need to ask clarifying questions, say what they currently think in case they’ve misunderstood what’s been said so far. Geoffrey Irving: And so there’s a broad space of different interaction protocols we’d like to explore that fix the holes in debate. And this is kind of building up towards that. One reason for doing GopherCite is if you want to do general debate, again, taking into account a bunch of leaf evidence. So debate is about machines and humans discussing some argument for something being true. But at the ends of that tree are leaves where you have checkable facts of some form. And directly, you can just have facts that every human knows, but that just isn’t that large a space of facts. And so this is extending the space of leaves so that we can implement more practical versions of debate on more interesting tasks. So I think these do fit...