Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Learning the prio, published by Paul Christiano on the AI Alignment Forum. Suppose that I have a dataset D of observed (x, y) pairs, and I’m interested in predicting the label y for each point x in some new set D. Perhaps D is a set of forecasts from the last few years, and D is a set of questions about the coming years that are important for planning. The classic deep learning approach is to fit a model f on D, and then predict y using f(x). This approach implicitly uses a somewhat strange prior, which depends on exactly how I optimize f. I may end up with the model with the smallest l2 norm, or the model that’s easiest to find with SGD, or the model that’s most robust to dropout. But none of these are anywhere close to the “ideal” beliefs of a human who has updated on D. This means that neural nets are unnecessarily data hungry, and more importantly that they can generalize in an undesirable way. I now think that this is a safety problem, so I want to try to attack it head on by learning the “right” prior, rather than attempting to use neural nets as an implicit prior. Warm-up 1: human forecasting If D and D are small enough, and I’m OK with human-level forecasts, then I don’t need ML at all. Instead I can hire a human to look at all the data in D, learn all the relevant lessons from it, and then spend some time forecasting y for each x. Now let’s gradually relax those assumptions. Warm-up 2: predicting human forecasts Suppose that D is large but that D is still small enough that a human can extract all the relevant lessons from it (or that for each x in D, there is a small subset of D that is relevant). In this case, I can pay humans to make forecasts for many randomly chosen x in D, train a model f to predict those forecasts, and then use f to make forecasts about the rest of D. The generalization is now coming entirely from human beliefs, not from the structural of the neural net — we are only applying neural nets to iid samples from D. Learning the human prior Now suppose that D is large, such that a human can’t update on it themselves. Perhaps D contains billions of examples, but we only have time to let a human read a few pages of background material. Instead of learning the unconditional human forecast P(y|x), we will learn the forecast P(y|x, Z), where Z is a few pages of background material that the human takes as given. We can also query the human for the prior probability Prior(Z) that the background material is true. Then we can train f(y|x, Z) to match P(y|x, Z), and optimize Z for: log Prior(Z) + sum((x, y) ~ D) log f(y|x, Z) We train f in parallel with optimizing Z, on inputs consisting of the current value of Z together with questions x sampled from D and D. For example, Z might specify a few explicit models for forecasting and trend extrapolation, a few important background assumptions, and guesses for a wide range of empirical parameters. Then a human who reads Z can evaluate how plausible it is on its face, or they can take it on faith in order to predict y given x. The optimal Z is then the set of assumptions, models, and empirical estimates that works best on the historical data. The human never has to reason about more than one datapoint at a time — they just have to evaluate what Z implies about each datapoint in isolation, and evaluate how plausible Z is a priori. This approach has many problems. Two particularly important ones: To be competitive, this optimization problem needs to be nearly as easy as optimizing f directly on D, but it seems harder: finding Z might be much harder than learning f, learning a conditional f might be much harder than learning an unconditional f, and jointly optimizing Z and f might present further difficulties. Even if it worked our forecasts would only be “human-level” in a fairly restrictive sense — they woul...