Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Newsletter #45, published by Rohin Shah on the AI Alignment Forum. Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter. Highlights Learning Preferences by Looking at the World (Rohin Shah and Dmitrii Krasheninnikov): The key idea with this project that I worked on is that the state of the world is already optimized for our preferences, and so simply by looking at the world we can infer these preferences. Consider the case where there is a vase standing upright on the table. This is an unstable equilibrium -- it's very easy to knock over the vase so it is lying sideways, or is completely broken. The fact that this hasn't happened yet suggests that we care about vases being upright and intact; otherwise at some point we probably would have let it fall. Since we have optimized the world for our preferences, the natural approach is to model this process, and then invert it to get the preferences. You could imagine that we could consider all possible reward functions, and put probability mass on them in proportion to how likely they make the current world state if a human optimized them. Basically, we are simulating the past in order to figure out what must have happened and why. With the vase example, we would notice that in any reward function where humans wanted to break vases, or were indifferent to broken vases, we would expect the current state to contain broken vases. Since we don't observe that, it must be the case that we care about keeping vases intact. Our algorithm, Reward Learning by Simulating the Past (RLSP), takes this intuition and applies it in the framework of Maximum Causal Entropy IRL (AN #12), where you assume that the human was acting over T timesteps to produce the state that you observe. We then show a few gridworld environments in which applying RLSP can fix a misspecified reward function. Rohin's opinion: In addition to this blog post and the paper, I also wrote a post on the Alignment Forum expressing opinions about the work. There are too many disparate opinions to put in here, so I'd recommend reading the post itself. I guess one thing I'll mention is that to infer preferences with a single state, you definitely need a good dynamics model, and a good set of features. While this may seem difficult to get, it's worth noting that dynamics are empirical facts about the world, and features might be, and there is already lots of work on learning both dynamics and features. Technical AI alignment Iterated amplification sequence Security amplification (Paul Christiano): If we imagine humans as reasoners over natural language, there are probably some esoteric sentences that could cause "failure". For example, maybe there are unreasonably convincing arguments that cause the human to believe something, when they shouldn't have been convinced by the argument. Maybe they are tricked or threatened in a way that "shouldn't" have happened. The goal with security amplification is to make these sorts of sentences difficult to find, so that we will not come across them in practice. As with Reliability amplification (AN #44), we are trying to amplify a fast agent A into a slow agent A that is "more secure", meaning that it is multiplicatively harder to find an input that causes a catastrophic failure. You might expect that capability amplification (AN #42) would also improve security, since the more capable agent would be able to notice failure modes and remove them. However, this would likely take far too long. Instead, we can hope to achieve security amplification by making reasoning abstract and explicit, with the hope that when reasoning is explicit it becomes harder to trigger the underlying failure mode, since you have to get your att...