Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fixing The Good Regulator Theorem , published by johnswentworth on the AI Alignment Forum. Conant & Ashby’s “Every Good Regulator Of A System Must Be A Model Of That System” opens with: The design of a complex regulator often includes the making of a model of the system to be regulated. The making of such a model has hitherto been regarded as optional, as merely one of many possible ways. In this paper a theorem is presented which shows, under very broad conditions, that any regulator that is maximally both successful and simple must be isomorphic with the system being regulated. (The exact assumptions are given.) Making a model is thus necessary. This may be the most misleading title and summary I have ever seen on a math paper. If by “making a model” one means the sort of thing people usually do when model-making - i.e. reconstruct a system’s variables/parameters/structure from some information about them - then Conant & Ashby’s claim is simply false. What they actually prove is that every regulator which is optimal and contains no unnecessary noise is equivalent to a regulator which first reconstructs the variable-values of the system it’s controlling, then chooses its output as a function of those values (ignoring the original inputs). This does not mean that every such regulator actually reconstructs the variable-values internally. And Ashby & Conant’s proof has several shortcomings even for this more modest claim. This post presents a modification of the Good Regulator Theorem, and provides a reasonably-general condition under which any optimal minimal regulator must actually construct a model of the controlled system internally. The key idea is conceptually similar to some of the pieces from Risks From Learned Optimization. Basically: an information bottleneck can force the use of a model, in much the same way that an information bottleneck can force the use of a mesa-optimizer. Along the way, we’ll also review the original Good Regulator Theorem and a few minor variants which fix some other problems with the original theorem. The Original Good Regulator Theorem We’re interested mainly in this causal diagram: The main goal is to choose the regulator policy P R X to minimize the entropy of outcome Z . Later sections will show that this is (roughly) equivalent to expected utility maximization. After explaining this problem, Conant & Ashby replace it with a different problem, which is not equivalent, and they do not bother to point out that it is not equivalent. They just present roughly the diagram above, and then their actual math implicitly uses this diagram instead: Rather than choosing a regulator policy P R X , they instead choose a policy P R S . In other words: they implicitly assume that the regulator has perfect information about the system state (and their proof does require this). Later, we’ll talk about how the original theorem generalizes to situations where the regulator does not have perfect information. But for now, I’ll just outline the argument from the paper. We’ll use two assumptions: The entropy-minimizing distribution of Z is unique (i.e. if two different policies P R S both achieve minimum entropy, they both produce the same Z -distribution). This assumption avoids a bunch of extra legwork which doesn’t really add any substance to the theorem. Z is a deterministic function of R S . Note that we can always make this hold by including any nondeterministic inputs to Z in S itself (though that trick only works if we allow R to have imperfect information about S , which violates Conant & Ashby’s setup. more on that later). The main lemma then says: for any optimal regulator P R S Z is a deterministic function of S . Equivalently: all R -values r with nonzero probability (for a given S -value s ) must give the same Z r s Intuitive argument: if t...