Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: chinchilla's wild implications, published by nostalgebraist on July 31, 2022 on The AI Alignment Forum. (Colab notebook here.) This post is about language model scaling laws, specifically the laws derived in the DeepMind paper that introduced Chinchilla. The paper came out a few months ago, and has been discussed a lot, but some of its implications deserve more explicit notice in my opinion. In particular: Data, not size, is the currently active constraint on language modeling performance. Current returns to additional data are immense, and current returns to additional model size are miniscule; indeed, most recent landmark models are wastefully big. If we can leverage enough data, there is no reason to train ~500B param models, much less 1T or larger models. If we have to train models at these large sizes, it will mean we have encountered a barrier to exploitation of data scaling, which would be a great loss relative to what would otherwise be possible. The literature is extremely unclear on how much text data is actually available for training. We may be "running out" of general-domain data, but the literature is too vague to know one way or the other. The entire available quantity of data in highly specialized domains like code is woefully tiny, compared to the gains that would be possible if much more such data were available. Some things to note at the outset: This post assumes you have some familiarity with LM scaling laws. As in the paper, I'll assume here that models never see repeated data in training. This simplifies things: we don't need to draw a distinction between data size and step count, or between train loss and test loss. I focus on the parametric scaling law from the paper's "Approach 3," because it's provides useful intuition. Keep in mind, though, that Approach 3 yielded somewhat different results from Approaches 1 and 2 (which agreed with one another, and were used to determine Chinchilla's model and data size). So you should take the exact numbers below with a grain of salt. They may be off by a few orders of magnitude (but not many orders of magnitude). 1. the scaling law The paper fits a scaling law for LM loss L, as a function of model size N and data size D. Its functional form is very simple, and easier to reason about than the L(N,D) law from the earlier Kaplan et al papers. It is a sum of three terms: The first term only depends on the model size. The second term only depends on the data size. And the third term is a constant. You can think about this as follows. An "infinitely big" model, trained on "infinite data," would achieve loss E. To get the loss for a real model, you add on two "corrections": one for the fact that the model's only has N parameters, not infinitely many one for the fact that the model only sees D training examples, not infinitely many Here's the same thing, with the constants fitted to DeepMind's experiments on the MassiveText dataset. plugging in real models Gopher is a model with 280B parameters, trained on 300B tokens of data. What happens if we plug in those numbers? What jumps out here is that the "finite model" term is tiny. In terms of the impact on LM loss, Gopher's parameter count might as well be infinity. There's a little more to gain on that front, but not much. Scale the model up to 500B params, or 1T params, or 100T params, or 3^^^3 params . . . and the most this can ever do for you is an 0.052 reduction in loss. Meanwhile, the "finite data" term is not tiny. Gopher's training data size is very much not infinity, and we can go a long way by making it bigger. Chinchilla is a model with the same training compute cost as Gopher, allocated more evenly between the two terms in the equation. It's 70B params, trained on 1.4T tokens of data. Let's plug that in: Much better! Without using any more compute, we...