Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Newsletter #27, published by Rohin Shah on the AI Alignment Forum. Dan Hendrycks has now joined, and will likely write summaries primarily on adversarial examples and robustness. As with Richard, his summaries are marked as such; I'm reviewing some of them now but expect to review less over time. Highlights 80K podcast with Paul Christiano (Paul Christiano and Rob Wiblin): This is a mammoth 4-hour interview that covers a lot of ground. I'll try to state the main points without the supporting arguments in roughly chronological order, listen to the podcast for more. - The problem of AI safety is that we don't know how to build AI that does what we want it to do. It arises primarily because each actor faces a tradeoff between AI systems being maximally effective at its task, and being robustly beneficial. - AI safety has had much more attention in the last few years. - Everyone agrees that we don't know how to build AI that does what we want, but disagrees on how hard the problem is, or how it should be framed. - The best arguments against working on alignment are opportunity cost (eg. working on biosecurity instead) and that the problem might be very easy or impossible, but even then it seems like work would be valuable for getting information about how hard the problem actually is. - It's not very important for the best AI safety team to work with the best ML team for the purpose of pursuing alignment research, but it is important for actually building powerful aligned AI. - The variance in outcomes from AGI come primarily from uncertainty in how hard the technical problem is, how people behave about AGI, and then how good we are at technical safety research. The last one is easiest to push on. - It seems useful to build organizations that can make commitments that are credible to outsiders. This would allow the top AI actors to jointly commit that they meet a particular bar for safety, though this would also require monitoring and enforcing to be effective, which is hard to do without leaking information. - We should expect slow takeoff, as Paul defines it. (I'm ignoring a lot of detail here.) - We should focus on short timelines because we have more leverage over them, but the analogous argument for focusing on fast takeoff is not as compelling. - Paul places 15% probability on human labor being obselete in 10 years, and 35% on 20 years, but doesn't think he has done enough analysis that people should defer to him. - Comparing current AI systems to humans seems like the wrong way to measure progress in AI. Instead, we should consider what we'd be able to do now if AI becomes comparable to humans in 10-20 years, and compare to that. - We can decompose alignment into the problem of training an AI given a smarter overseer, and the problem of creating a sufficiently smart overseer. These roughly correspond to distillation and amplification respectively in IDA. (There's more discussion of IDA, but it should be pretty familiar to people who have engaged with IDA before.) Reactions fall into three camps: a) IDA is hopelessly difficult, b) IDA is focusing on far-away problems that will be easy by the time they are relevant, and c) optimistic about IDA. - Very few people think about how to solve the full problem, that is, solve alignment in the limit of arbitrarily intelligent AI. MIRI doesn't think about the question because it seems obviously doomed to them, while the broader ML community wants to wait until we know how to build the system. The other approaches are debate (AN #5), which is very related to IDA, and inverse reinforcement learning (IRL). However, there are key problems with IRL, and research hasn't shed much light on the core of the problem. - AI safety via debate also shares the insight of IDA that we can use AI to help us define a better training ...