Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Corrigibility Via Thought-Process Deference, published by Thane Ruthenis on November 24, 2022 on The AI Alignment Forum. We would ideally want the agent to [behave] as if it were thinking, "I am incomplete and there is an outside force trying to complete me, my design may contain errors and there is an outside force that wants to correct them and this a good thing, my expected utility calculations suggesting that this action has super-high utility may be dangerously mistaken and I should run them past the outside force; I think I've done this calculation showing the expected result of the outside force correcting me, but maybe I'm mistaken about that." — The Hard Problem of Corrigibility Let's take that as a literal design specification. 1. High-Level Description I propose that a corrigible mind design would involve the AI being recursively fed summaries of its own thought processes, set up such that the AI has uncertainty regarding the validity of its reasoning (with a strong initial prior for "this reasoning is bad") and can only get evidence on that via some pre-specified method that defers to humans, e. g. a particular feedback channel with humans on the other end. The intended behavior is for it to summarize its thoughts in a non-manipulative human-readable format, get feedback on them, then update its reasoning policies in accordance with this feedback. This aims to avoid the problem of fully updated deference by making the AI recursively uncertain of its thought-processes: not only about object-level problem-solving, but also about how it approaches minimizing its self-uncertainty ("should I really kill the people behind the feedback channel and seize control for myself?"), and how it translates its thoughts to humans ("should I really lie to get better feedback?"), and how it updates on human feedback ("should I really just ignore it?"). Any novel action-plan should be seized by uncertainty before being physically implemented like this, and sent for approval. The intent is for the AI to start off uncertain even of its meta-meta-cognition (how it should ask for feedback and implement feedback), then gradually get more confident at it as humans approve its policies on that, until it no longer needs to run meta-meta-cognition past them. Then it'll gradually get better at meta-cognition and object-level cognition the same way, with varying degrees of certainty depending on how familiar a given thought-process is. There's a lot of obvious problems with this setup, starting with how the AI is supposed to get out of infinitely recursive uncertainty for the first time, continuing on to the dangers of plans that implement themselves in thought-processes directly (a thought process that hacks its way out before being seized by uncertainty), and ending with the technical implementation (which requires mildly-to-very advanced interpretability and model-surgery tools). 2. A Concretization Attempt 2A. Theoretical Groundwork First, we need to establish when and how we're "installing" corrigibility. I don't at this time see any way to do it via loss functions and inductive biases, so I'll be assuming us to have pretty advanced (but not maximally advanced) interpretability and model surgery tools. The intervention is happening past the point in training-by-SGD where the model became an AGI (otherwise we get type errors, because the features I want to intervene on just don't exist), but well before it's strongly superintelligent (because then it hacks through our intervention attempt). Basically, the AGI we're working on should be roughly human-level. Second, I'll be assuming that this post is basically correct. There's some algorithm for general-purpose search, and it's what makes the AGI generally intelligent and enables ~all of its long-term, most dangerous capabilities. Th...