Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The longest training run, published by Jaime Sevilla on August 17, 2022 on The AI Alignment Forum. In short: Training runs of large Machine Learning systems are likely to last less than 14-15 months. This is because longer runs will be outcompeted by runs that start later and therefore use better hardware and better algorithms. ScenarioLongest training runHardware improvements3.55 yearsHardware improvements + Software improvements1.22 yearsHardware improvements + Rising investments3.12 monthsHardware improvements + Rising investments + Software improvements2.52 months Larger compute budgets and a better understanding of how to effectively use compute (through, for example, using scaling laws) are two major driving forces of progress in recent Machine Learning. There are many ways to increase your effective compute budget: better hardware, rising investments in AI R&D and improvements in algorithmic efficiency. In this article we investigate one often-overlooked but plausibly important factor: how long—in terms of wall-clock time—you are willing to train your model for. Here we explore a simple mathematical framework for estimating the optimal duration of a training run. A researcher is tasked with training a model by some deadline, and must decide when to start their training run. The researcher is faced with a key problem: by delaying the training run, they can access better hardware, but by starting the training run soon, they can train the model for longer. Using estimates of the relevant parameters, we calculate the optimal training duration. We then explore six additional considerations, related to 1) how dollar-budgets for compute rise over time, 2) the rate at which algorithmic efficiency improves, 3) whether developers can upgrade their software over time 4) what would happen in a more realistic framework with stochastic growth, 5) whether it matters for the framework that labs are not explicitly optimizing for optimal training runs and 6) what would happen if they rent instead of buy hardware. Our conclusion depends on whether the researcher is able to upgrade their hardware stack while training. If they aren't able to upgrade hardware, optimal training runs will likely last less than 3 months. If the researcher can upgrade their hardware stack during training, optimal training runs will last less than 1.2 years. These numbers are likely to be overestimates, since 1) we use a conservative estimate of software progress, 2) real-world uncertainty pushes developers towards shorter training runs and 3) renting hardware creates an incentive to wait for longer and paralellize the run close to the deadline. ScenarioLongest training runHardware improvements3.55 yearsHardware improvements + Software improvements1.22 yearsHardware improvements + Rising investments3.12 monthsHardware improvements + Rising investments + Software improvements2.52 months A simple framework for training run lengths Consider a researcher who wants to train a model by some deadline T. The researcher is deciding when to start the training run in order to maximize the amount of compute per dollar. The researcher is faced with a key trade-off. On one hand, they want to delay the run to access improved hardware (and/or other things like larger dollar-budgets and better algorithms.) On the other hand, a delay reduces the wall-clock time that the model is trained for. Suppose that hardware price-performance is increasing as follows: where H0 is the initial FLOPS/$ and gH is the rate of yearly improvement. If we start a training run at time S, the cumulative FLOP/$ at time t≥S will be equal to: where H(S) is the price-performance of the available hardware when we start our run (in FLOP/$/time), and (t−S) is the amount of time since we started our run. Given a fixed dollar-budget, when should we buy...