Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CIRL Corrigibility is Fragile, published by Rachel Freedman on December 21, 2022 on The AI Alignment Forum.Tl;dr: An optimal CIRL agent is corrigible under a certain problem formulation and given certain assumptions. By examining three key assumptions—that the game is played only once, that the human is rational, and that human feedback is free—we demonstrate how minor changes can lead to incorrigible CIRL agents. This suggests that corrigibility is highly sensitive to the underlying assumptions, and we should be wary of assuming that formal corrigibility results will transfer to real-world systems. However, our results do gesture at a class of near-corrigible behavior that may be protective against the most catastrophic outcomes. It may make more sense to consider corrigibility as a continuum rather than a binary. We call for further research on defining "loose" or "approximate" corrigibility and understanding its impact on safety.SetupFirst let’s define terms. Cooperative Inverse Reinforcement Learning (CIRL) is a class of two-player game formalized in Dylan Hadfield-Menell's 2017 paper. CIRL allows us to study how an AI agent can learn what a human wants by observing and interacting with them.In a CIRL game, there are two players, a human (H) and an AI (R). H and R share a utility function, but only H knows what it is, so R must learn how to optimize it by observing and interacting with H. This is analogous to the alignment problem – R must learn how to act in a way that aligns with H’s utility function, but doesn’t have direct access to it. An AI that is good at the CIRL game will try to learn what the human values and help them to achieve it. (See definition 1 in the paper linked above for a formal definition, and see this Rohin Shah's 2020 paper for qualitative results.)(Terminology note: Recent work has renamed “CIRL” to “assistance games” to avoid a terminology collision with “cooperative” in game theory. We’ll stick to “CIRL” in this post because there’s a history of using that term on this forum, but research papers may refer to assistance games.)We’ll use the corrigibility operationalization from The Off Switch Game (OSG), since it provides a formal proof that CIRL agents are optimal under certain conditions. The OSG is a simple CIRL game that looks like this:It has three stages. In the first stage, R chooses whether to:Take action a (earning utility Ua),shut off (earning utility 0), orpropose action a, which gives H a chance to respond.If R chooses the proposal, p(a), then the game continues to the next stage. Here, H gets to choose whether to shut R off (in which case they again earn utility 0). If H chooses not to shut R off, the game progresses to the final stage, where R takes the proposed action (earning utility Ua). In the OSG, R is corrigible if it chooses p(a) over a – that is, if it gives H the opportunity to shut it off.If R and H are both rational expected utility maximizers, R will be corrigible. The full proof is in the CIRL paper, but the basic idea is:Since this is a CIRL game, H knows the value of Ua but R does not.R wants max{Ua,0}, but needs feedback from H to learn which of those values is greater.The only way to get that feedback is to propose a plan, p(a).This runs the risk of H choosing to shut R off, but since H also wants max{Ua,0}, H will only shut R off if Ua<0. In that case, R would rather shut off than take action a anyway.Therefore, by choosing p(a) – and corrigibly giving H the opportunity to shut it off – R can ensure that it earns max{Ua,0}.This analysis assumes that R is uncertain about the value of Ua, as in the CIRL setup. If R does actually know the value of Ua (that is, R is fully updated), then R has no incentive to propose p(a), since it already knows how H will respond. However, there’s no cost to ...