Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Response to Holden’s alignment plan, published by Alex Flint on December 22, 2022 on The AI Alignment Forum.This work was supported by the Monastic Academy for the Preservation of Life on Earth. You can support my work directly here.I will give a short presentation of this work followed by discussion on Saturday (Dec 24) at 9am Pacific / 12pm Eastern. RSVP here.OutlineThis is a response to Holden Karnofsky’s plan for aligning powerful AI systems in the case that such systems are developed soon.I give a summary of Holden’s plan, and then critique it.My basic critique is that the tools Holden is proposing are too similar to what has been tried in the construction of human institutions, and I argue that we have failed to create large numbers of human institutions that alleviate rather than exacerbate existential risk.The Karnofsky PlanHolden has written about how we might align very powerful AI systems if such systems were to be developed soon. The basic approach Holden proposes is to train AI systems to be honest, legible, and corrigible, and value-aligned using human supervision. Honesty means not engaging in deceit or manipulation of the supervisor. Legibility means creating artifacts that help supervisors to accurately review decisions. Corrigibility means being willing to stop when the human supervisor says no, no matter how strongly the AI believes that it is doing the right thing. Value-alignment means pursuing the goals of the supervisor. To train such AI systems, Holden says we need to pay attention to the following properties of the training environment:(1) The human supervisor should provide high-quality feedback. This means that if we are training an AI system to be honest then the human supervisor should give a positive reward if and only if the AI system is, in fact, being honest. It can be hard to know whether an AI system is being honest. Therefore we may design further tools to help the human assess whether the AI system is being honest. In order to do this we may train auxiliary AI systems to assess our primary AI system, or we may restrict the complexity of the primary AI systems, or we may set up an ecosystem of differently-trained AI systems in such a way that dishonesty is "weeded out" by ecological dynamics.(2) The AI learner should correctly generalize what it has been taught. This means that if we are training an AI system to be honest then the AI system should be honest even when presented with situations very unlike those that it encountered during training. To accomplish this we may identify, during training, which situations are most likely to cause undesirable behavior given the current state of the AI, and then subject the AI to those situations and provide negative rewards until the undesirable behavior is corrected. We may train auxiliary AI systems to help with identifying the situations most likely to cause undesirable behavior given the current state of the AI (adversarial training).(3) The AI learner should not break out of the training container. This means that if we are training an AI system to be honest, it should be possible for the human supervisor to hide certain things outside the training container, such as schemes for testing the AI system in clever ways. If the AI system escapes the training container, hacks the human supervisor’s cell phone, and listens to the human supervisors discussing the structure of the next test, then the AI system may be able to appear honest without actually being honest. More generally, the AI system should not escape the training container during training. To accomplish this we may build auxiliary AI systems that help to find and patch security holes in the software that holds the training container.(4) The AI learner should be thoroughly tested before deployment. This means that af...