Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Abram Demski's ELK thoughts and proposal - distillation, published by Rubi on July 19, 2022 on The AI Alignment Forum. This post was written for the SERI MATS program. I thank Evan Hubinger and Leo Gao for their mentorship in the program. Further thanks go to Evan Hubinger (again), Simon Marshall, and Johannes Treutlein for specific comments regarding the content of this post. The Eliciting Latent Knowledge (ELK) problem was first introduced by Paul Christiano, Marx Xu, and Ajeya Cotra. Subsequently, Abram Demski worked on the problem, collected his thoughts in a thought dump post, and won a prize for his proposal “use the reporter to define causal interventions on the predictor”. Here I attempt to clarify and flesh out these thoughts in order to make them more accessible. I assume familiarity with ELK, but not with Abram’s post. Very little of this post is my own original content. Epistemic status: 60% confident that I am accurately representing Abram’s thoughts at the time he wrote his post, 75% confident that I am representing them accurately enough not to change the key takeaways, 80% confident that the extended proposals and counterexamples I propose are logically sound. Introduction When the technical report for Eliciting Latent Knowledge (ELK) was first released, it was followed by a contest offering substantial cash prizes for proposed solutions. The contest wrapped up with thirty-two prizes awarded, split into six categories. In the prize announcement, Paul and Mark noted that there was substantial convergence among proposals, but in two of the six categories there was only a single submission. Abram Demski’s proposal was the sole entry in the category “Use reporter to define causal interventions”. A quick summary is provided in the prize results post, but the full context and justification for it is better provided in a longer post by Abram. Specifically, Abrams shares his thoughts on how the ELK problem ties to philosophical concepts of truth and knowledge, and how that can be used to inform solutions. Evan Hubinger describes Abram’s ELK work as “some of the most exciting work on the problem right now”, but Abram’s post is self-described as a “thought dump”, and the prize post does not explicitly lay out the case for how the proposal would work. I aim to re-organize, clarify, and flesh-out his work in order to make it more accessible for forum readers. I will go in the opposite order of Abram’s original post, starting with the prize-winning proposal and its counter-examples before going into the context and motivation behind it, then ending with a discussion on possible modifications and extensions. The Proposal and Its Counterexamples The Proposal The direct reporter must have a deep understanding of how the predictor works, and some human simulators will have this understanding too. This means that these reporters know not just how the state of the predictor would look when processing an existing data point, but also how it would look under hypothetical data points. A human could then ask the reporter to search over potential data points and provide the configuration of the predictor under counterfactual scenarios. This ability to get these scenarios from the reporter gives us a method for tripping up human simulators. We start with the predictor’s state and the answers to a set of questions from the reporter in a real scenario, then ask the reporter to suggest the smallest change to the predictor’s state that would make the reporter give a particular different answer to a specified question in the new state. The reason the proposal is called defining causal interventions is because causal interventions will be the ones with the smallest impact on the predictor’s state, changing only elements causally downstream of the intervention. The reporter is then ...