Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Externalized reasoning oversight: a research direction for language model alignment, published by tamera on August 3, 2022 on The AI Alignment Forum. Summary In this post, I’ll present a research direction that I’m interested in for alignment of pretrained language models. TL;DR: Force a language model to think out loud, and use the reasoning itself as a channel for oversight. If this agenda is successful, it could defeat deception, power-seeking, and other forms of disapproved reasoning. This direction is broadly actionable now. In recent publications, prompting pretrained language models to work through logical reasoning problems step-by-step has provided a boost to their capabilities. I claim that this externalized reasoning process may be used for alignment if three conditions are met: We develop trustworthy tests to check if the reported reasoning is authentic (complete, straightforward, and causally responsible for the output). We train models that pass the tests of reasoning authenticity. We oversee the externalized reasoning to detect reasoning of which we would not approve, and either filter the resulting actions or fine-tune against reasoning approval. If these conditions hold, we should be able to detect and avoid models that reason through convergent instrumental goals to be deceptive, power-seeking, non-myopic, or reason through other processes of which we don’t approve. Reasoning oversight should provide stronger guarantees of alignment than oversight on model outputs alone, since we would get insight into the causally responsible reasoning process that gave rise to a certain output. I believe that productive research can be done today toward all three of these conditions. I’m doing exploratory work into this strategy as part of my SERI MATS project in the large language model alignment stream. Background I’ll use “externalized reasoning” to refer to a prompting strategy for a pretrained language model that has it read in a question, produce a number of steps of reasoning, and then produce an answer to the question as its final output. Some examples of this include chain of thought prompting [1, 2], selection-inference [3], and scratchpad reasoning [4]. These types of strategies have recently led to state of the art results on a number of logical reasoning benchmark tasks, most notably 50.3% accuracy on the MATH dataset [5]. A few examples of chain of thought reasoning can be found here. Future work may develop externalized reasoning approaches different from those that currently exist; ideally, the research directions described here will be able to generalize to sufficiently similar approaches. Research directions First condition: assess reasoning authenticity My main focus right now is in developing tests that can be used to assure that the model’s reasoning is authentic. What “authenticity” means is currently vaguely defined, but fundamentally I want it to encompass whatever is needed to trust that disapproved reasoning couldn’t go undetected. Some aspects of this might include: Causal responsibility: The displayed reasoning isn’t post-hoc, but is causally responsible for the conclusion that the model came to. Completeness: No steps of reasoning are missing, given the level of detail at which the reasoning is presented (which should be an appropriate level for oversight). Straightforwardness: Hidden messages aren’t steganographically encoded in the reasoning. It's useful to note that we have read and write access to the externalized reasoning process, which is useful for creating tests that probe causal responsibility. I’ve written up my current test ideas in another document, but I’d like to avoid that document being crawled by a web scraper which adds it to a language model’s training corpus. Accordingly I’ve put together a poor man’s CAPTCHA which...