Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Builder/Breaker for Deconfusion, published by Abram Demski on September 29, 2022 on The AI Alignment Forum. This is something of a grab-bag of thoughts I've had about the Builder/Breaker game. The ELK document had a really nice explanation of its research methodology in terms of an imaginary dialogue between a "Builder" who makes positive proposals, and a "Breaker" who tries to break them. To an extent, this is just the ordinary philosophical method, and also a common pattern in other research areas. However, I felt that the explicit write-up helped to clarify some things for me. We might think of the Builder/ Breaker game as an adversarial game where either the builder or breaker "wins", like AI debate. However, I find it more fruitful to think of it as a cooperative game. When the game is played by AI safety researchers, the players have a common goal of finding robust plans to avoid catastrophic outcomes. The builder/breaker game merely organizes cognitive work: both Builder and Breaker are trying to map the space of proposals, but each takes primary responsibility for avoiding a different kind of error (false positives vs false negatives). Security Mindset I think Builder/Breaker is a good way to understand Eliezer's notion of security mindset (1, 2). The Builder is trying to construct a positive argument for safety, with (at least) the following good properties: The argument clearly states its assumptions. Each assumption is as plausible as possible (because any grain of doubt indicates a possibility of failure). There are as few assumptions as possible (because more assumptions mean more ways the plan can fail). Each step of reasoning is sound. The conclusion of the argument is a meaningful safety guarantee. I will call such a plan robust. We can question whether AI safety research should focus on robust plans. I won't dwell on this question too much. Clearly, some endeavors require robust plans, while others do not. AI safety seems to me like a domain which requires robust plans. I'll leave it at that for now. In any case, coming up with robust plans has proven difficult. The Builder/Breaker game allows us to incrementally make progress, by mapping the space of possibilities and marking regions which won't work. Example: Wireheading I could easily forgive someone for reading a bunch of AI alignment literature and thinking "AI alignment researchers seem confident that reinforcement learners will wirehead.". This confusion comes from interpreting Breaker-type statements as confident predictions. (Someone might try to come up with alignment plans which leverage the fact that RL agents wirehead, which imho would be approximately as doomed as a plan which assumed agents wouldn't. Breaker start saying "What if the agent doesn't wirehead?" instead of "What if the agent wireheads?".) Reward is not the optimization target. The point isn't that RL agents necessarily wirehead. The point is that reinforcement signals cannot possibly rule out wireheaders. This is an example of a very important class of counterexamples. If we are trying to teach an agent some class of behaviors/beliefs using feedback, the feedback may be consistent with what we are actually trying to teach, but it will also be consistent with precisely modeling the feedback process. A model which understands the feedback process in detail, and identifies "maximizing good feedback" as the goal, will plausibly start trying to manipulate that feedback. This could mean wireheading, human manipulation, or other similar strategies. In the ELK document, the "human simulator" class of counterexamples represents this failure mode. Since this is such a common counterexample, it seems like any robust plan for AI safety needs to establish confidently that this won't occur. (It also happens that we have empirical eviden...