Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Low-stakes alignment, published by Paul Christiano on the AI Alignment Forum. Right now I’m working on finding a good objective to optimize with ML, rather than trying to make sure our models are robustly optimizing that objective. (This is roughly “outer alignment.”) That’s pretty vague, and it’s not obvious whether “find a good objective” is a meaningful goal rather than being inherently confused or sweeping key distinctions under the rug. So I like to focus on a more precise special case of alignment: solve alignment when decisions are “low stakes.” I think this case effectively isolates the problem of “find a good objective” from the problem of ensuring robustness and is precise enough to focus on productively. In this post I’ll describe what I mean by the low-stakes setting, why I think it isolates this subproblem, why I want to isolate this subproblem, and why I think that it’s valuable to work on crisp subproblems. 1. What is the low-stakes setting? A situation is low-stakes if we care very little about any small number of decisions. That is, we only care about the average behavior of the system over long periods of time (much longer than the amount of time it takes us to collect additional data and retrain the system). For example, this requires that all of the AI systems in the world can’t corrupt the training process quickly or seize control of resources from humans. If they try, we can keep collecting data and fine-tuning them, and this will cause their behavior to change before anything irreversibly bad happens. For a more formal definition see section 6. 2. Why do low stakes require only outer alignment? If the stakes are low, we can train our model on the decisions that actually arise in practice rather than needing to anticipate tricky decisions in advance. Moreover, because the payoff from an individual action is always small, we can focus on average-case performance and achieve reasonable sample complexities without any additional tricks. The main substantive claim is that we don’t need to worry about the “distributional shift” between past decisions and future decisions. When the distribution of inputs change, the system may behave poorly for a while, but if we keep retraining on the new data then it will eventually adapt. If individual decisions are low stakes, then the total cost of all of this adaptation is small. I give this argument in more detail in section 7. Formally this resembles an online regret bound (this textbook gives a nice introduction to online learning). SGD satisfies such a bound in the case of convex losses. For messy model classes like neural networks we usually can’t prove much interesting about SGD (either for online or offline learning), but for a variety of reasons I think it’s reasonable to expect a similar online bound. I discuss this in more detail in section 8. This isn’t to say that we can totally ignore optimization difficulties, or the online nature of the problem. But it appears that the main difficulty is constructing a good enough objective and arguing that it is sufficiently easy to optimize. 3. Why focus on this subproblem first? I think it’s really great to focus on a good subproblem if you can find one. If you solve your subproblem, then you’ve made progress. If you get stuck, well then you were probably going to get stuck anyway and at least you’re stuck on something easier. When working on a big problem like alignment, I feel like it’s easy to bounce off of every solution because it doesn’t handle the whole problem immediately, and splitting into subproblems is a key way to get over that failure. I think that the low-stakes setting is a particularly good and clean subproblem: it’s definitely not harder than the original, there are clear ways in which it’s much easier, and solving it would represent real prog...