Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Newsletter, published by Rohin Shah on the AI Alignment Forum. Highlights Introducing the AI Alignment Forum (FAQ) (habryka): The Alignment Forum has officially launched! It aims to be the single online hub for researchers to have conversations about all the ideas in the field, while also helping new researchers get up to speed. While posting is restricted to members, all content is cross-posted to LessWrong, where anyone can engage with it. In addition, for the next few weeks there will be a daily post from one of three new sequences on embedded agency, iterated amplification, and value learning. Rohin's opinion: I'm excited for this forum, and will be collating the value learning sequence for its launch. Since these sequences are meant to teach some of the key ideas in AI alignment, I would probably end up highlighting every single post. Instead of that, I'm going to create new categories for each sequence and summarize them each week within the category, but you should treat them as if I had highlighted them. Reinforcement Learning with Prediction-Based Rewards (Yuri Burda and Harri Edwards) (summarized by Richard): Researchers at OpenAI have beaten average human performance on Montezuma's Revenge using a prediction-based curiosity technique called Random Network Distillation. A network with fixed random weights evaluates each state; another network with the same architecture is trained to predict the random network's output, given its input. The agent receives an additional reward proportional to the predictor's error on its current state. The idea behind the technique is that the predictor's error will be higher on states different from those it's been trained on, and so the agent will be rewarded for exploring them. This paper follows from their study on curiosity (AN #20) in which a predictor was trained to predict the next state directly, and the agent was rewarded when its error was high. However, this led to high reward on states that were unpredictable due to model limitations or stochasticity (e.g. the noisy TV problem). By contrast, Random Network Distillation only requires the prediction of a deterministic function which is definitely within the class of functions representable by the predictor (since it has the same architecture as the random network). Richard's opinion: This is an important step forward for curiosity-driven agents. As the authors note in the paper, RND has the additional advantages of being simple to implement and flexible. Technical AI alignment Embedded agency sequence Embedded Agents (Abram Demski and Scott Garrabrant): This post introduces embedded agency, which refers to the notion of an "agent" that is more realistic than the version considered in mainstream AI, which is best formalized by AIXI. An embedded agent is one that is actually a part of the environment it is acting in, as opposed to our current AI agents which model the environment as external to them. The problems around embedded agency fall into four main clusters, which future posts will talk about. Rohin's opinion: This post is a great summary of the sequence to come, and is intuitive and easy to understand. I strongly recommend reading the full post -- I haven't summarized it much because it already is a good summary. Decision Theory (Abram Demski and Scott Garrabrant): The major issue with porting decision theory to the embedded agency section is that there is no longer a clear, well-defined boundary between actions and outcomes, such that we can say "if I take this action, then this outcome occurs". In an embedded setting, the agent is just another part of the environment, and so if the agent is reasoning about the environment, it can also reason about itself, and its reasoning can tell it something about what its actions will be. But if you know what ...