Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open Problems with Myopia, published by Mark Xu, Evan Hubinger on the AI Alignment Forum. Thanks to Noa Nabeshima for helpful discussion and comments. Introduction Certain types of myopic agents represent a possible way to construct safe AGI. We call agents with a time discount rate of zero time-limited myopic, a particular instance of the broader class of myopic agents. A prototypical example is a time-limited myopic imitative agent. In theory, such an agent has some desirable safety properties because a human would only take safe actions (although any imperfect imitation would be unsafe). Since the agent is time-limited myopic, it will never imitate poorly now to make it easier to imitate easier later. For example, it would never give a human a simple plan so it could more easily imitate the human executing the plan. We might run into issues if the agent intends to myopically imitate humans but guesses incorrectly. Such an agent might witness a human purchasing paperclips, infer that humans tend to acquire paperclips, and proceed to convert the universe into paperclips. This agent would not be safe because it is not . Myopia does not contribute to capability robustness; we only hope it helps create intent aligned agents. In particular, . One way of viewing deception is as sacrificing reward now for reward later, which suggests that time-limited myopia should prevent it. However, there are several ways time-limited myopia fails to rule out deceptive alignment. What we mean by myopia is myopic cognition, which is distinct from myopic training. Myopic training might produce myopic cognition, but . It is currently unclear precisely what myopic cognition is. We hope a proper characterization of myopic cognition will resolve the problems presented. Following , we use the term “reward” for the thing given to the agent by the training environment and the term “utility” for the thing that agent is internally trying to maximize. Open Problems We present a useful toy environment for reasoning about deceptive alignment. In this environment, there is a button. Agents have two actions: to press the button or to refrain. If the agent presses the button, they get +1 reward for this episode and -10 reward next episode. One might note a similarity with the traditional marshmallow test of delayed gratification. In some sense, agents that do not press the button will engage in deception; both agents trade reward now for more reward later. We are interested in creating agents that robustly do not press the button. We suppose that our agents are reward-maximizers, i.e. they get utility equal to the reward given by the training environment. Superrationality[1] If an agent at time T could trade with agents other time steps, they might make a deal: "I will not press the button if and only if T-1 does not the button." Accepting this deal gives the agent +9 utility in that particular time step, so even time-limited myopic agents would agree to this deal. However, there are many ways that trade could happen. In particular, agents' reasoning processes will be similar between time steps, and the agents themselves will know this, know they know this, etc. Since agents at any particular timestep are likely that they have predecessors, they can be reasonably confident that their predecessors would benefit them if they decide to benefit their successors. However, if they press the button, they can be confident their predecessors also pressed the button. Thus not pressing the button would give +9 utility. Superrationality thus poses a problem for time-limited myopia. Even agents that would never sacrifice utility now for utility later might accept this reasoning and press the button. Applied to deception, there is ambiguity about what utility is. The obvious candidate is "existence"; failing to be d...