Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Concrete Proposal for Adversarial IDA, published by Evan Hubinger on the AI Alignment Forum. Note: This post came out of a conversation with Geoffrey Irving and Buck Shlegeris. Epistemic Status: I suspect Paul has already thought of most or all of the ideas presented here, though I nevertheless found the exercise of carefully specifying an IDA implementation helpful and suspect others may find reading it helpful as well. This is a proposal for how to train a machine learning model to approximate HCH using Iterated Distillation and Amplification (IDA). This particular proposal came out of a desire to use a debate-like adversary to improve the amplification process, and the primary goal of this proposal is to show how one could do that. Though I have tried to retain a lot of the relevant detail, I have made two simplifications to make this proposal easier to specify: I am attempting to approximate something closer to weak HCH rather than strong HCH and I am only allowing the generation of two subquestions at a time. I am confident that those simplifications could easily be dropped, though I think doing so here would only make this presentation more complicated. Before I proceed, I want to make one final note: this is not a proposal for how to build an aligned AGI. I think there are still a whole bunch of issues that would prevent this proposal from actually working. Definitions We will start with some initial definitions: Let Q be the set of all questions in natural language. Let A be the set of all answers in natural language. Let M be the sum type of either Q × Q or A representing either an answer to the given question or two subquestions to help answer it. Let H Q → A be the answer that a human gives to the given question. Let H fan out Q → M be the answer or subquestion pair generated by a human when asked what to do with the given question. Let H fan in Q × Q × A × Q × A → M be the answer or two subquestions generated by a human to some question when given answers to two subquestions related to that question. Let M L Q → Δ A be a model (the training procedure for which we will describe below) from questions to a probability distribution over strings representing answers. Specifically, we will implement the probability distribution by having our model output an embedding vector which yields the probability distribution when fed into some trained language model (by repeatedly conditioning on previous characters and multiplying all the conditional probabilities). Let M L fan out Q → Δ M be the function from questions to an embedding vector representing a distribution over answers or subquestions generated by asking M L what to do with the given question. For the two subquestion case we enforce that the distribution is symmetric wrt interchange of the subquestions. Let M L fan in Q × Q × A × Q × A → Δ M be the function from two answers to an embedding vector representing a distribution over answers or subquestions generated by asking M L to integrate the given subquestion answers into an answer to the original question. We again enforce symmetry wrt interchange of the subquestions. Let Adv Q × A → R be an adversary model which we will train (as described below) to predict how good the given answer is to the given question. Let Adv fan out Q × M → R be an adversary model for M L fan out generated by calling Adv using the same transformation as used to implement M L fan out from M L Let Adv fan in Q × Q × A × Q × A × M → R be an adversary model for M L fan in generated by calling Adv using the same transformation as used to implement M L fan in from M L Let ϵ be some Gumbel random variable such that each use of ϵ below is its own independent sample from a Gumbel distribution. We use a Gumbel random variable so that we can use the Gumbel-max trick to effectively do Bol...