Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Less Threat-Dependent Bargaining Solutions?? (3/2), published by Diffractor on August 20, 2022 on The AI Alignment Forum. In the previous two posts, we went over various notions of bargaining. The Nash bargaining solution. The CoCo value. Shapley values. And eventually, we managed to show they were all special cases of each other. The rest of this post will assume you've read the previous two posts and have a good sense for what the CoCo value is doing. Continuing from the last post, the games which determine the payoff everyone gets (not to be confused with the games that directly entail what actions are taken) are all of the form "everyone splits into two coalitions S and N/S, and both coalitions are trying to maximize "utility of my coalition - utility of the opposite coalition"". Now, in toy cases involving two hot-dog sellers squabbling over whether to hawk their wares at a beach or an airport, this produces acceptable results. But, in richer environments, it's VERY important to note that adopting "let's go for a CoCo equilibria" as your rule for how to split gains amongst everyone incentivizes everyone to invent increasingly nasty ways to hurt their foes. Not to actually be used, mind you. To affect how negotiations go. After all, if you invent the Cruciatus curse, then in all those hypothetical games where your coalition faces off against the foe coalition, and everyone's utility is "the utility of my coalition - the utility of the foe coalition"... well, you sure can reduce the utility of the foe coalition by a whole lot! And so, your team gets a much higher score in all those games. Of course, these minimax games aren't actually played. They just determine everyone's payoffs. And so, you'd end up picking up a whole lot of extra money from everyone else, because you have the Cruciatus curse and everyone's scared of it so they give you money. In the special case of a two-player game, getting access to an option which you don't care about and the foe would pay $1000 to avoid, should let you demand a 500$ payment from the foe as a "please don't hurt me" payment. But Which Desiderata Fails? Step 1 in figuring out how to get an outcome which is Not That is to look at the list of nice properties which the CoCo solution uniquely fulfills, and figure out which one to break. As it turns out from picking through the paper, the assumption that must be broken is the axiom of "gaining access to more actions shouldn't lead to you getting less value". As a quick intuitive way of seeing why it should fail, consider the game of Chicken. If you physically can't swerve (because your car started off not having a steering wheel), and are locked into always going straight through no fault of your own, then any sensible opponent will swerve and you will get good payoffs. Adding in the new option of being able to swerve means the opponent will start demanding that you swerve sometimes, lowering your score. As a rough intuition for how the "the CoCo value is the only way to fulfill these axioms" proof works, it is reasoning as follows: "Hm, let's say I only had access to my minimax action, that maximized minb(U1(a,b)−U2(a,b)). It makes me relatively much better off than my foe. Any sensible foe going up against this threat would simply press a button that'd just give us both the CoCo value, instead of playing any other action in response. By the axiom of "adding more options can't hurt me", I can add in all my other actions and not get less money. And then by the axiom of "adding redundant actions doesn't affect anything", I can take away the foe's CoCo value button and nothing changes. And so, in this game against the foe, I have to get a value equal or greater to my CoCo value. But the foe can run through this same reasoning from their side, and so in this game, we must both get th...