Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Eliciting Latent Knowledge (ELK) - Distillation/Summary, published by Marius Hobbhahn on June 8, 2022 on The AI Alignment Forum. This post was inspired by the AI safety distillation contest. It turned out to be more of a summary than a distillation for two reasons. Firstly, I think that the main idea behind ELK is simple and can be explained in less than 2 minutes (see next section). Therefore, the main value comes from understanding the specific approaches and how they interact with each other. Secondly, I think some people shy away from reading a 50-page report but I expect they could get most of the understanding from reading/skimming this summary (I'm aware that the summary is longer than anticipated but it's still a >5x reduction of the original content). I summarized the ELK report and the results of the ELK competition. I personally think ELK is very promising and therefore want to make extra sure that the summary is readable and accurate. In case anything is unclear or wrong, please let me know. Context and Introduction Eliciting Latent Knowledge was written by Paul Christiano, Ajeya Cotra and Mark Xu. Before getting to the main summary, I want to provide some background and context. Additional to the summary of the main report, I provide a summary of the results from the ELK prize contest in the end. My aim is to give a brief summary of “everything related to ELK” with links to the long versions. Paul Christiano has been working on various projects in AI safety before founding the Alignment Research Center (ARC). He lead the LLM alignment team at OpenAI and is known for, among other contributions, AI safety via debate and Iterated Distillation and Amplification (IDA). From my very limited outside perspective and some of his podcast appearances (e.g. at 80K or the AI x-risks podcast), Paul Christiano seems to come back to similar problems with many current alignment techniques. Whether you choose inverse RL, AI safety via debate or IDA, there is always the risk that sufficiently powerful models create problems that can only be solved by understanding what the model “truly thinks”--by eliciting latent knowledge (ELK). Of course, other people want to understand what’s going on in Neural Networks (NNs) as well. Chris Olah, for example, has successfully worked on technical NN interpretability for many years now (see e.g. circuits) and there are many other approaches to NN transparency in academia. To my understanding, there are two main differences between ELK and these other approaches. Firstly, ELK operates on a different scale. NN interpretability often looks at low-level neurons or circuits (which gets messy when your model has Billions of parameters) while ELK aims to generate natural language answers or descriptions that reflect the model's true beliefs as if it “had drunk a truth serum”. In the optimal case, we don’t need to understand the model in detail, the model just pro-actively tells us what it believes on its own. Secondly, ARC approaches ELK more theoretically than is common in technical alignment research. Conventionally, technical alignment researchers come up with a theory, then test it in practice, find the flaws and improve it. ARC, on the other hand, skips the practical implication for now and tries to find theoretical counterexamples for their current proposals. Once they found a counterexample, they try to come up with a new proposal that fixes the problem and then iterate. In the best case, they arrive at a proposal for which they can’t find any major theoretical flaws and then implement that. Summary of the ELK report I will try to stay close to the structure and naming patterns that the ELK report uses. In my summary, I will necessarily remove some nuance and detail. However, I still want it to be always true. Therefore, if you think som...