Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Intermittent Distillations #2, published by Mark Xu on the AI Alignment Forum. Servant of Many Masters: Shifting priorities in Pareto-optimal sequential decision-making (Andrew Critch and Stuart Russell) Servant of Many Masters: Shifting priorities in Pareto-optimal sequential decision-making Summary A policy (over some partially observable Markov decision process (POMDP)) is Pareto optimal with respect to two agents with different utility functions if it is not possible to construct a policy that achieves higher utility for one of the agents without doing worse for the other agent. A result by Harsanyi shows that for agents that have the same beliefs, Pareto optimal policies act as if they are maximizing some weighted sum of the two agents' utility functions. However, what if the agents have different beliefs? Interestingly, if two agents disagree about the world, it is possible to construct policies that are better for both of them from that agent's perspective. For example, suppose that Alice and Bob are deciding how to split a cake. Suppose also that the cake is either red or green. Alice believes that the cake is red with 0.9 and Bob believes the cake is green with 0.9. A policy that says "If the cake is red, give it to Alice. If the cake is green, give it to Bob." will be viewed favorably by both of them. In fact, the sum of the utility Alice expects to get and the utility Bob expects to get is greater than can be achieved by any policy maximizing a weighted linear combination of their two utility functions. Intuitively, when Alice and Bob both agree to the conditional policy, they're betting against each other about empirical facts about the world. More specifically, Alice can be viewed as offering to bet Bob that the cake is red, which Bob readily accepts. In this way, the conditional policy ties the expected utility of the two agents to previously irrelevant facts about their world models, giving them both higher expected utility from their perspectives. The key result of the paper shows that all Pareto-optimal policies will have an implicit "bet settling" mechanism. One way of thinking about this is that since any difference in empirical beliefs can produce positive-sum bets between agents, a Pareto-optimal policy must implicitly make all such bets between those agents. Loosely speaking, the result shows that any policy that is Pareto-optimal with respect to a collection of agents will maximize a weighted linear combination of "how much does this agent benefit" and "how well did this agent predict empirical observations." Since Harsanyi assumes the agents have the same beliefs about the world, the second component is identical for all agents, so Harsanyi's theorem is a special case of the authors' result. The result implies that if a contract between parties is designed to be Pareto-optimal, it will tend to "settle bets" amongst empirical beliefs of those parties (provided they have different beliefs.) The authors suggest making this "bet settling" explicit might improve contract efficiency and transparency. Opinion A perspective I've been developing recently is something along the lines of "betting is fundamental." For instance, no Dutch book arguments pin down Bayes' rule as the proper update formula (given ). If you relax the Dutch books to those that are efficiently computable, you get Logical Induction (sorta). Abram Demski writes in : It is a truth more fundamental than Bayes' Law that money will flow from the unclever to the clever. This paper represents another place where betting arises out of seemingly unrelated considerations. I also appreciate the frequent intuitive motivation of the result. I have a slight lingering confusion about how the assumption that agents have knowledge about other agents' beliefs interacts with Aumann's Agreement theore...