Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A distillation of Evan Hubinger's training stories (for SERI MATS), published by Daphne W on July 18, 2022 on The AI Alignment Forum. This post is a distillation of Evan Hubinger's post "how do we become confident in the safety of a machine learning system?", made as part of the summer 2022 SERI MATS program. While I have attempted to understand and extrapolate Evan's opinions, this post has not been vetted. Likewise, I use training stories (and contribution stories) to describe the methodology of proposals for safe advanced AI without the endorsement of those proposals' authors and based on a relatively shallow understanding of those proposals (due to my inexperience and time constraints). The opinions presented in this post are my own unless otherwise noted. Epistemic status: Exploratory Some day, all too soon from now, people will deploy the AI that seals humanity's fate. There are many scenarios for how this comes about or what happens afterward. Some have multiple AI negotiating or fighting for dominance, others one. Some think the handoff of power from humans to AI will go slowly, others fast. But whatever the case, there is one question that these people need to get right: "Is the AI we're about to deploy safe?" For people to get this answer right when it matters, two things need to happen: we need tools to accurately determine whether an advanced AI is safe, and we need an advanced AI those tools approve of that we can deploy in time for it to matter. Training stories are a tool to evaluate proposals for making this happen. Specifically, they're meant to analyze complete proposals for training prosaic advanced AI: advanced AI which has similar enough architecture to current systems that most research is cross-applicable. The reason to focus on complete proposals is that it gets us to pay attention to what will be important when push comes to shove. From there, we can backpropagate and get an estimate of what research paths are most beneficial. Evan thinks theoretical prosaic alignment is the most beneficial in expectation for him and many others to pursue, though he supports a broad spectrum of research. Because I think the original post describes the central concept of training stories very well, I have avoided retreading the same ground, and instead focused on expanding its usability. I will start with some more case studies of using training stories to analyze proposals, and then expand on the original post with contribution stories: a method for analyzing research agendas by how they factor into training stories. Training stories Case study: Imitative amplification + intermittent oversight In this case study, Evan's comments in An overview of 11 proposals for building safe advanced AI map well to the training story evaluation, and I don't feel like much was missing. However, the training story itself makes it more clear to me what the training goal might look like, which could help point at potential experiments with toy models that can be done today. This demonstrates how training stories can be used to clarify a path towards working AI. Training goal specification: The goal is to end up with a model M which accurately answers whether a model similar to M is safe to run, which suggests possible tests to determine if that model is safe to run, and which answers questions about some other topic T. This can take several forms, but let's try to work one out:M can't take all weights of M as input for inspection, so M has to look at sections of a model and share a summary of its conclusions with other M's to form a reliable answer. This could be organized like HCH. Since M is trained to imitate the output of Amp(M), which is prototypically a human consulting M and some selection of transparency tools, these summaries would be human-legible.Mechanically, M woul...