Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reliability amplification, published by Paul Christiano on the AI Alignment Forum. In a recent post I talked about capability amplification, a putative procedure that turns a large number of fast weak agents into a slower, stronger agent. If we do this in a naive way, it will decrease reliability. For example, if. Our weak policy fails with probability 1%. In order to implement a strong policy we combine 10 decisions made by weak agents. If any of these 10 decisions is bad, then so is the combination. .then the combination will be bad with 10% probability. Although the combination can be more powerful than any individual decision, in this case it is much less reliable. If we repeat policy amplification several times, our failure probability could quickly approach 1, even if it started out being exponentially small. Complementary to capability amplification is reliability amplification: given a policy H that usually works, can we implement a policy H that works significantly more reliably? To be slightly less imprecise (but still quite crude): Given a distribution A over policies that ε-close to a benign policy for some ε ≪ 1, can we implement a distribution A⁺ over policies which is δ-close to a benign policy of similar capability, for some δ ≪ ε? When I say that A is ε-close to a benign policy, I mean that there is a benign distribution over policies which has total variation distance at most ε from A. We can view this is as a simple mixture between a single aligned policy that is chosen with probability (1−ε) and an adversarial policy that is chosen with probability ε. I will move freely between talking about “policies” and “distributions over policies;” we can turn a distribution into a single policy by averaging. The notion of “implement” here is exactly the same as for capability amplification: we can use any number of implementations of A and whatever other resources are at our disposal, and must compute A⁺’s responses to a sequence of observations, within some time limit such as a day. Reliability amplification via deliberation I think there is one very natural way to do reliability amplification — if some variant of this idea doesn’t work, I’m not sure what else to try. As a warm-up, let’s consider the setting where the agents want to make a single decision rather than a sequence of decisions — we’ll return to the sequential case soon. We sample three agents A, B, and C from the given distribution. We ask them to talk for a while and then each propose an action. Then we ask them to do approval voting amongst the three options, and randomly pick from the maximally-approved-of proposals. If all three agents are aligned, then this proposal is clearly going to produce an aligned action. If two the agents behave adversarially then we are clearly out of luck, but that happens only with probability O(ε²), so we can basically ignore it. The interesting case is when there is one adversarial agent and two aligned agents. If we threw up our hands and picked randomly in this case, then we would have a failure probability of ε. So the question is: can we do better than that? The obvious hope is that aligned agents can do better-than-chance at recognizing each other / at recognizing good actions. If they can get any non-negligible advantage, then we can get a non-negligible reduction in error probability (and by iterating the procedure we can potentially get large reductions). The sequential case So far we have talked about making a single decision. The problem is more challenging when we need to make a sequence of decisions. We can try to simply generalize the voting approach, running a new vote for each action. To see the difficulty, suppose that the optimal policy looks as follows: Generate some hard-to-transfer internal state X. Think of X as something like an intuitiv...