Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Learning the prior and generalization, published by Evan Hubinger on the AI Alignment Forum. This post is a response to Paul Christiano's post “Learning the prior.” The generalization problem Generally, when we train models, we often end up deploying them in situations that are distinctly different from those they were trained under. Take, for example, GPT-3. GPT-3 was trained to predict web text, not serve as a dungeon master—and the sort of queries that people present to AI dungeon are quite different than random web text—but nevertheless GPT-3 can perform quite well here because it has learned a policy which is general enough that it continues to function quite effectively in this new domain. Relying on this sort of generalization, however, is potentially quite troublesome. If you're in a situation where your training and deployment data are in fact independently and identically distributed (i.i.d.), you can produce all sorts of nice guarantees about the performance of your model. For example, in an i.i.d. setting, you know that in the limit of training you'll get the desired behavior. Furthermore, even before the limit of training, you know that validation and deployment performance will precisely track each other such that you can bound the probability of catastrophic behavior by the incidence of catastrophic behavior on the validation data. In a generalization setting, on the other hand, you have no such guarantees—even in the limit of training, precisely what your model does off-distribution is determined by your training process's inductive biases. In theory, any off-distribution behavior is compatible with zero training error—the only reason machine learning produces good off-distribution behavior is because it finds something like the simplest model that fits the data. As a result, however, a model's off-distribution behavior will be highly dependent on exactly what the training process's interpretation of “simpler” is—that is, its inductive biases. And relying on such inductive biases for your generalization behavior can potentially have catastrophic consequences. Nuances with generalization That being said, the picture I've painted above of off-distribution generalization being the problem isn't quite right. For example, consider an autoregressive model (like GPT-3) that's just trained to learn a particular distribution. Then, if I have some set of training data X train and a new data point x ′ , there's no test you can do to determine whether x ′ was really sampled from the same distribution as X train . In fact, for any X train and x ′ ∉ X train , I can always give you a distribution that X train could have been sampled from that assigns whatever probability I want to x ′ . Thus, to the extent that we're able to train models that can do a good job for i.i.d. x ′ ∉ X train that is, that assign high probability to x ′ it's because there's an implicit prior there that's assigning a fairly high probability to the actual distribution you used to sample the data from rather than any other of the infinitely many possible distributions (this is the no free lunch theorem). Even in the i.i.d. case, therefore, there's still a real and meaningful sense in which your performance is coming from the machine learning prior. It's still the case, however, that actually using i.i.d. data does give you some real and meaningful guarantees—such as the ability to infer performance properties from validation data, as I mentioned previously. However, at least in the context of mesa-optimization, you can never really get i.i.d. data thanks to fundamental distributional shifts such as the the very fact that one set of data points is used in training and one set of data points is used in deployment. Paul Christiano's RSA-2048 example is a classic example of how that sort of fundam...