Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Newsletter #32, published by Rohin Shah on the AI Alignment Forum. Remember, treat all of the "sequence" posts as though I had highlighted them! Highlights Spinning Up in Deep RL (Joshua Achiam): OpenAI has released an educational resource aimed to help software engineers become skilled at deep reinforcement learning. It includes simple implementations of many deep RL algorithms (as opposed to the relatively complex, highly optimized implementations in Baselines), educational exercises, documentation, and tutorials. OpenAI will host a workshop on the topic at their headquarters on Feb 2nd, and are also planning to hold a workshop at CHAI some time in early 2019. Rohin's opinion: I know that a lot of effort has gone into this project, and I expect that as a result this is probably the best educational resource on deep RL out there. The main other resource I know of is the deep RL bootcamp, which probably supplements this resource nicely, especially with the lectures (though it is a year out of date). Technical AI alignment Embedded agency sequence Embedded World-Models (Abram Demski and Scott Garrabrant): A few slides have been added to this post since my summary last week, going into more detail about the grain-of-truth problem. This problem is particularly hard because your learned world model must include the world model itself inside of it, even in the presence of an environment that can behave adversarially towards the world model. It is easy to construct deterministic paradoxes where the world model cannot be correct -- for example, in rock-paper-scissors, if your model predicts what the opponent will do and plays the action that wins against the prediction, the opponent will (if they can) predict that and play the action that beats your action, falsifying your model. While game theory solves these sorts of scenarios, it does so by splitting the agent away from the environment, in a way that is very reminiscent of the dualistic approach. Recently, reflective oracles were developed, that solve this problem by having probabilistic models that were robust to self-reference, but they still assume logical omniscience. Subsystem Alignment (Abram Demski and Scott Garrabrant): Any agent is likely to be built out of multiple subsystems, that could potentially have their own goals and work at cross-purposes to each other. A simple unrealistic example would be an agent composed of two parts -- a world model and a decision algorithm (akin to the setup in World Models (AN #23)). The decision algorithm aims to cause some feature of the world model to be high. In this case, the decision algorithm could trick the world model into thinking the feature is high, instead of actually changing the world so that the feature is high (a delusion box). Why not just build a monolithic agent, or build an agent whose subcomponents are all aligned with each other? One reason is that our agent may want to solve problems by splitting into subgoals. However, what then prevents the agent from optimizing the subgoal too far, to the point where it is no longer helps for the original goal? Another reason is that when we make subagents to solve simpler tasks, they shouldn't need the whole context of what we value to do their task, and so we might give them a "pointer" to the true goal that they can use if necessary. But in that case, we have introduced a level of indirection, which a previous post (AN #31) argues leads to wireheading. Perhaps the most insidious case is search, which can produce subagents by accident. Often, it is easier to solve a problem by searching for a good solution than deriving it from first principles. (For example, machine learning is a search over functions, and often outperforms hand-designed programs.) However, when an agent searches for a good solution, the so...