Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Newsletter #33, published by Rohin Shah on the AI Alignment Forum. Find all Alignment Newsletter resources here. In particular, you can sign up, or look through the database of all summaries. One correction to last week's newsletter: the title Is Robustness at the Cost of Accuracy should have been Is Robustness the Cost of Accuracy. Highlights Reward learning from human preferences and demonstrations in Atari (Borja Ibarz et al): We have had lots of work on learning from preferences, demonstrations, proxy rewards, natural language, rankings etc. However, most such work focuses on one of these modes of learning, sometimes combined with an explicit reward function. This work learns to play Atari games using both preference and demonstration information. They start out with a set of expert demonstrations which are used to initialize a policy using behavioral cloning. They also use the demonstrations to train a reward model using the DQfD algorithm. They then continue training the reward and policy simultaneously, where the policy is trained on rewards from the reward model, while the reward model is trained using preference information (collected and used in the same way as Deep RL from Human Preferences) and the expert demonstrations. They then present a lot of experimental results. The main thing I got out of the experiments is that when demonstrations are good (near optimal), they convey a lot of information about how to perform the task, leading to high reward, but when they are not good, they will actively hurt performance, since the algorithm assumes that the demonstrations are high quality and the demonstrations "override" the more accurate information collected via preferences. They also show results on efficiency, the quality of the reward model, and the reward hacking that can occur if you don't continue training the reward model alongside the policy. Rohin's opinion: I'm excited to see work that combines information from multiple sources! In general with multiple sources you have the problem of figuring out what to do when the sources of information conflict, and this is no exception. Their approach tends to prioritize demonstrations over preferences when the two conflict, and so in cases where the preferences are better (as in Enduro) their approach performs poorly. I'm somewhat surprised that they prioritize demos over preferences, since it seems humans would be more reliable at providing preferences than demos, but perhaps they needed to give demos more influence over the policy in order to have the policy learn reasonably quickly. I'd be interested in seeing work that tries to use the demos as much as possible, but detect when conflicts happen and prioritize the preferences in that situation -- my guess is that this would let you get good performance across most Atari games. Technical AI alignment Embedded agency sequence Embedded Agency (full-text version) (Scott Garrabrant and Abram Demski): This is the text version of all of the previous posts in the sequence. Iterated amplification sequence The Steering Problem (Paul Christiano): The steering problem refers to the problem of writing a program that uses black-box human-level cognitive abilities to be as useful as a well-motivated human Hugh (that is, a human who is "trying" to be helpful). This is a conceptual problem -- we don't have black-box access to human-level cognitive abilities yet. However, we can build suitable formalizations and solve the steering problem within those formalizations, from which we can learn generalizable insights that we can apply to the problem we will actually face once we have strong AI capabilities. For example, we could formalize "human-level cognitive abilities" as Hugh-level performance on question-answering (yes-no questions in natural language), online lea...