Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exploring Mild Behaviour in Embedded Agents, published by Megan Kinniment on June 27, 2022 on The AI Alignment Forum. Thanks to Tristan Cook, Emery Cooper, Nicolas Macé and Will Payne for discussions, advice, and comments. All views are my own. I think this post is best viewed as an effort to point at a potentially interesting research direction. The specific arguments I make centered around inducing mild behaviour in embedded agents are rough and exploratory. Summary If an embedded agent's utility function is heavily dependent on the parts of the environment which constitute its own thinking processes, we might see behaviour that does not have an equivalent in unembedded agents. (This seems pretty likely to me. Of all the claims here I am most confident of this one.) We might be able to design measures/utility functions that constrain the parts of the environment that constitute the agent’s thinking process itself, such that we can moderate the strength of the optimisation that an embedded agent exerts on the rest of the environment. (This seems in theory possible to me but I am currently very uncertain about whether this is feasible in practice.) Following this, we may be able to induce mild optimisation in EU maximising embedded agents through the specification of the agent’s utility function, and this induced ~mild optimisation could help protect that very same mildness-inducing specification from being gamed. (Similar thoughts to above) Measures that impose optimisation pressure on the thinking process might be more robust than usual to specification problems if they can induce some degree of mild optimisation even when they are not specified in a water-tight manner. To the extent that we can manage to specify a mediocre efficiency measure that induces even some small degree of mild optimisation, it will also slightly reduce the optimisation pressure the agent puts on the specification as a whole. In this way, there might be some potential for measures like these to be made somewhat self-protecting against specification problems like edge instantiation and nearest unblocked strategies. (Again, similar thoughts to above, though I do currently find this general idea promising.) Context A fundamental property of embedded agents is that the agent’s thinking process is a part of the environment itself (for example, some process running on a physical chip). This can be a driver of certain ‘bad’ behaviours like wireheading, unwanted self-preservation, or extreme resource acquisition for the sake of self-improvement. However, perhaps we can also use the fact that embedded agents have physically instantiated ‘thinking processes’ to try and induce safer behaviour. Two major things that make alignment so difficult are: Avoiding small degrees of misspecification of ‘what we actually want’ is very difficult The specification is likely to be put under an enormous amount of optimisation pressure by a powerful optimiser (which is likely to ‘find’ and ‘blow up the consequences’ of any small misspecification) These considerations motivate work on approaches such as designing low-impact measures or methods of implementing mild optimisation. Low-impact measures mostly address the first point, by aiming to heuristically specify part of ‘what we actually want’ whilst also reducing the consequences of misspecifications. Mild optimisation mostly addresses the second point, by aiming to ‘tone down’ the optimisation pressure. Embedded agents have physical instantiations of the optimisation processes that they use to decide how to optimise their environment (e.g. some algorithm running on a physical chip). What would happen if we attempted to construct a utility function that strongly incentivises the agent to constrain the physical instantiation of its own optimising process? Might it be...