Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Synthesizing amplification and debate, published byEvan Hubinger on the AI Alignment Forum. Background One possible way to train an amplification model is to use an auxiliary reinforcement learning objective to help guide the training of the amplification model. This could be done either by training two separate models, an agent and a question-answerer, or a single model trained on a joint objective. For example, from a comment Paul left on “A dilemma for prosaic AI alignment:” I normally imagine using joint training in these cases, rather than pre-training + fine-tuning. e.g., at every point in time we maintain an agent and a question-answerer, where the question-answerer "knows everything the agent knows." They get better together, with each gradient update affecting both of them, rather than first training a good agent and then adding a good question-answerer. (Independently of concerns about mesa-optimization, I think the fine-tuning approach would have trouble because you couldn't use statistical regularities from the "main" objective to inform your answers to questions, and therefore your question answers will be dumber than the policy and so you couldn't get a good reward function or specification of catastrophically bad behavior.) In my last post, I expressed skepticism of such non-imitative amplification approaches, though in this post I want to propose a possible way in which some of my concerns with this style of approach could addressed by integrating ideas from AI safety via debate. I'll start by describing the basic idea in broad terms, then give a more careful, technical description of the sort of training procedure I have in mind. The proposal The basic idea is as follows: debate naturally yields an RL objective, so if you want to add an auxiliary RL objective to amplification, why not use the RL objective from debate? Specifically, the idea is to conduct a debate not between copies of the model M , but between copies of the amplified model Amp M (where Amp M is a human with access to the model M ). That gives you both an RL reward arising from the winner of the debate as well as a supervised loss from the difference between the model's output and the amplified model's output. That's the basic proposal—now I'll try to give a more careful, technical description of how I think it should go. Definitions We will start with some initial definitions. Let Q be the set of all questions in natural language. Let A be the set of all answers in natural language. Let Q ∈ Δ Q be the training distribution of natural language questions. Let H Q → A be a human. Let M Q → A be a model. Let Amp H M Q H “What answer would you give to Q given access to M ?” Let Approx H M Q if Samp Q M Q then H Q else M Q where Samp Q × A → B is some means of stochastically determining whether it is worth it to call H or not.[1] Let Amp ≈ H M Amp Approx H M M Let M t a r g e t Q → A be a target network updated regularly to match M Training procedure Then, consider the following training procedure. Generate some question Q ∼ Q Conduct a debate according to the following procedure while building up a history S and starting at t 0 Let arg t Amp ≈ H M “What is the next argument you would give on Q in context S ?” Append to the history S the contents of arg t as well as a trace of what happened in calculating arg t , including all the necessary information to use transparency tools to inspect what happened. Let continue t Amp ≈ H M t a r g e t “Is arg t a sufficient answer to Q in context S ?” If not continue t , increment t and return to step 1 in the debate. Let winner Amp ≈ H M t a r g e t “Were the even arguments in S more helpful than the odd arguments in answering Q ?” Train M on the combination of the debate-style RL reward derived from winner r 1 for the winner and r − 1 for the loser)...