Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pavlov Generalizes, published by Abram Demski on the AI Alignment Forum. Sarah Constantine recently wrote about the Pavlov strategy for iterated prisoner's dilemma. She points out that it deserves to be better-known, given that it has similar virtues to Tit-for-Tat (TfT). I strongly agree. Much of my thinking about puzzles in decision theory such as cooperation/coordination/bargaining and logical causality (through the lense of logical time) has been shaped by an assumption that TfF type behavior is "correct" in some sense, and needs to be appropriately generalized. These intuitions have been formed without an awareness of Pavlov, and so, are worth reconsidering. It turns out that Pavlov-type behavior is much easier to generalize that TfT. Pavlov-esque strategies only require seeing when things are going well or poorly for you. TfT-esque strategies require an agent to understand what cooperation vs defection means in a particular game. This requires the agent to understand the wider rules of the game (not just the payoff on a given round), locate the other agents involved (distinguishing them from the environment), infer the values of the other agents, define a set of common cooperative norms which can be coordinated on, and figure out whether other agents have followed them! Several of these elements are both difficult in practice and difficult to define in principle. In this post, I would like to explain how to generalize Pavlov to a wide set of games (achieving some surprisingly strong, but not actually very good, coordination properties), and also convey some intuitions about Pavlov-style vs TfT-style coordination. Generalizing Pavlov The paper Achieving Pareto Optimality Through Distributed Learning by Marden, Young, and Pao describes a strategy which allows players in an iterated game to eventually converge to a Pareto-optimal action profile with high probability. In fact, the strategy described converges to action profiles which maximize the sum of all the utilities! That should sound too good to be true. Some big caveats are coming. The strategy is, roughly, as follows. Agents have two different states: content and discontent. When content, the agent copies its own action from the previous round, whatever it was. When discontent, the agent plays randomly. An agent flips between content and discontent based on how well things are going for it. There is a parameter, experimentation rate (e.r.), which determines how likely an agent is to flip states. Discontent: Discontent agents look at how good the previous round was for them. They change to "content" based on the payoff of the previous round and the e.r.: the higher the payoff and the higher the e.r., the more likely they are to become content. A low e.r. makes them very picky, remaining discontent the majority of the time, but much less likely to become content after seeing low payoffs, and only moderately less likely after high payoffs. Content, consistent payoff: When content, the agent's behavior depends on whether the payoff is the same as the twice-previous round. If the previous and twice-previous rounds had the same payoffs, content agents remain content, and are very likely to take the same action as on the previous round. They take a different action at the exploration rate. Content, inconsistent payoff: If the previous round had a payoff not consistent with the twice-previous round, a content agent is likely to become discontent (remaining content with probability increasing in the payoff and the e.r.). The nice property this strategy has is that for a low e.r., players end up eventually taking an efficient set of actions with high probability. It accomplishes this without agents knowing anything about each other; everyone acts on the payoffs they receive alone. This is kind of nice, but looking at...