Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Searching for Consequentialist Structure, published by leogao on March 27, 2022 on The AI Alignment Forum. Editor’s note: I’m experimenting with having a lower quality threshold for just posting things even while I’m still confused and unconfident about my conclusions, but with this disclaimer at the top. Thanks to Kyle and Laria for discussions. In “Searching for Bayes Structure”, Eliezer argues that things which arrive at the truth must be doing something Bayes-like somewhere in there, even if it’s extremely inefficient or accidental or filtered through some process that looks very different from the outside. If it has no Bayes structure, it literally cannot arrive at the truth. I think it’s plausible that Bayes is to epistemic rationality as consequentialism is to instrumental rationality. Here, by consequentialism I mean the ability to get to some goal from a broader set of initial states, which is how I understood Elliezer’s use of the term in the dialogues. It does not have to be an EUM/VNM-rational. Importantly, to be good at consequentialism comes with all the stuff like instrumental convergence (and consequently self preservation, power seeking, etc). In other words, there’s some minimal amount of consequentialism you need to achieve a given goal, and there’s no getting around this lower bound. Of course, some agents do a lot more consequentialism than is necessary, but I interpret some people (i.e Eliezer) as arguing that even this lower bound is too much consequentialism for, say, performing a pivotal action, because it’s fundamentally very difficult to do. As an analogy, if your goal is to buy an apple for $1, any plans which fail to collect at least $1 will simply not be able to acquire an apple, and even though there also exist plans which collect $100 (which is more dangerous than collecting $1), if even collecting $1 is too dangerous by default, then even lobotomizing a $100-collecting agent to only collect $1 is scarcely comforting. The main lesson I draw from this is that if we want to do something consequentialist then we have to somehow figure out how to align consequentialists, and that making the agent worse at consequentialism isn’t a solution. I think it also seems productive to figure out where the lower bound of how much consequentialism we need (for, i.e a pivotal action) and the upper bound of how much consequentialism is safe are relative to each other. I think there's also a way in which this applies to tool AI type arguments as well. Ultimately, the reason we want tools is they help us accomplish some goal (for instance, performing a pivotal action). So, we can view the entire human+tool system as an agent that is trying to get to some goal, where there's some human oversight to ensure that the plans pass muster. This system, to get to goals, will have to do consequentialism. While having the human provide the glue between the model and the real world does add some safety, I would argue that this is approximately as safe as just humans vetting plans from an agent, which is itself not really sufficient because many plans which seem non suspicious to us are actually dangerous in some unforeseen way (diving into whether this is most plans or vanishingly few plans seems like a useful avenue - on the one hand one would think that making a plan sound useful while still doing something bad might require a lot of optimization, on the other hand "I can't see anything going wrong with this" is a hugely lower bar than "nothing will go wrong with this" resulting in most plans that satisfy the former not satisfying the latter). While the human providing the "outer loop" so to speak does allow the human a little bit more control in that they have control over the particular reasoning processes used to arrive at the action, I don't think this is a ver...