Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Two Challenges for ELK, published by derek shiller on February 21, 2022 on The AI Alignment Forum. This post sketches two challenges to ARC's project around eliciting latent knowledge that differ somewhat in kind from the challenges ARC is most concerned about. They relate to the difficulty in distinguishing beliefs from other representations. Introduction The problem of ELK, as outlined in ARC's technical report, is to figure out: given an AI trained to complete a task (the 'task AI'), how can we design a second AI (the 'reporter AI') that can answer questions about how the task AI understands the world. If we can know what the task AI is thinking, we can better assess whether its decisions are dangerous. The central challenge identified by ARC to solving the problem of ELK concerns the possibility that the reporter AI will not learn to answer questions using what the task AI believes (e.g. answering honestly), but instead answer questions using what a human observer might think is the case on the basis of the available evidence (answering empathetically). We train the reporter AI to answer based on the facts as we understand them (and know the task AI to understand them) in relatively simple cases. We hope that the reporter will answer questions in more complex cases based on how the task AI understands the world, but it might instead answer based on what it thinks a human would mistakenly believe. If the human questioner is ignorant of some complex details that the task AI knows, the reporter AI might choose to leave them out to tell a simpler story that will satisfy the questioner. It is not obvious how to train an AI to answer honestly rather than empathetically. While this is a significant challenge, I don't think it is the only challenge for ELK. In thinking through the problem, I became worried about several others. I here describe two other challenges. Though they are somewhat speculative, ARC aims for a solution to the problem of ELK that deals with the worst case, and these challenges fall under that umbrella. Even if we figure out how to make an honest rather than a empathetic reporter, these challenges will remain. 1.) The Right Representation Challenge We do not know how the task AI might encode the the things that it believes, but what it knows about the world must somehow be reflected in its neural activity. For simplicity, let's suppose that the task AI utilizes explicit independent representational states that are encoded in its network in something vaguely like the way sentences are written in a book. The reporter AI would need to figure out where in the network these sentences are recorded. It would then have to figure out how to interpret the sentences and use them to provide answers to questions in English. The reporter AI would also need to figure out which sentences recorded in the task AI's network are really believed and which sentences are recorded for other reasons. The task AI might produce a variety of representations that it doesn't fully believe (i.e. doesn't straightforwardly rely on to make decisions) for various purposes. If we just scan the network for sentences, we may find sentences that it doesn't quite believe. These may still correlate with what it does believe in training scenarios, and so a reporter AI might learn to use them to answer questions. If the reporter latches on to those sentences, then it might give misleading answers once outside of training. Consider the following possibilities: The task AI makes predictions using the simulated mental states of various agents. In the central case of the report, the task is to prevent a thief from stealing a diamond -- the task AI might keep track of what the thief believes is happening, what a security guard thinks is happening, what other would-be thieves might later believe h...