Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is part two of: Forecasting TAI with biological anchors, published by Ajeya Cotra. Part 2: How training data requirements scale with parameter count This report emerged from discussions with our technical advisors Dario Amodei and Paul Christiano. However, it should not be treated as representative of either of their views; the project eventually broadened considerably, and my conclusions are my own. This is a work in progress and does not represent Open Philanthropy’s institutional view. We are making it public to make it easier to gather feedback, to help inform others’ thinking in the effective altruism community, and to allow for follow-on work outside of Open Phil. However, we may edit it substantially in the future as we gather feedback from a broader audience and investigate open questions. Accordingly we have not done an official publication or blog post, and would prefer for now that people not share it widely in a low-bandwidth way (e.g., just posting key graphics on Facebook or Twitter). This report has been split into four Google docs in order to load faster. This is Part 2; the first part is here, the third part is here, and the fourth part is here. Additional materials (collected in this folder): Quantitative model: the Python notebook Biological anchor hypotheses for 2020 training computation requirements; a template spreadsheet When required computation may be affordable; and my best guess, conservative, and aggressive forecasts. Supplemental materials: a document containing various appendices; a folder of figures for the report; the spreadsheet Extrapolations of data and compute to train models; and the Python notebook Compute price trends, which draws on data in this folder. In Part 1, I provided an overview of the framework and estimates, provided definitions for key abstractions used in the model, and generated an estimate for the number of FLOP / subj sec of a transformative model. In Part 2, I will provide an overview of the evidence we can use to think about training data requirements for a transformative model, by generating a scaling law relating the number of parameters that characterizes a model to the number of data points required to train it. The scaling law derived in this part will be used in Part 3 to estimate 2020 training computation requirements for the Genome Anchor hypothesis and the Neural Network hypotheses. Specifically, the amount of computation required to train a transformative model for each of those hypotheses will be estimated as Train FLOP = (F FLOP / subj sec) x (T subj sec of training) I generated an probability distribution over F, the FLOP / subj sec of a transformative model, in Part 1 here; my median estimate is ~1e16 FLOP / subj sec. Here, I will focus on how to extrapolate T, the number of subjective seconds of data that a model must be trained on, as a function of the parameter count P of a transformative model -- the larger the parameter count, the more data is needed to train the model. The Neural Network hypotheses and Genome Anchor hypothesis estimate P differently; this will be covered in Part 3. In the rest of Part 2, I will: Explain why machine learning theory would predict that data requirements tend to scale linearly with parameter count (more). Examine two papers which ran controlled experiments attempting to elicit scaling laws relating dataset size and model parameter count, both of which conclude that dataset size D scales can be described as a power law function of parameter count P (i.e. D KP), with the more relevant result predicting that this scaling is sub-linear (more). Introduce the concept of a “horizon length” to operationalize what “one data point” means for different ML problems (more). Present some estimates of the number of data points various recent RL models were trained on, which is con...