Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Environments as a bottleneck in AGI development, published by Richard Ngo on the AI Alignment Forum. Given a training environment or dataset, a training algorithm, an optimiser, and a model class capable of implementing an AGI (with the right parameters), there are two interesting questions we might ask about how conducive that environment is for training an AGI. The first is: how much do AGIs from that model class outperform non-AGIs? The second is: how straightforward is the path to reaching an AGI? We can visualise these questions in terms of the loss landscape of those models when evaluated on the training environment. The first asks how low the set of AGIs is, compared with the rest of the landscape. The second asks how favourable the paths through that loss landscape to get to AGIs are - that is, do the local gradients usually point in the right direction, and how deep are the local minima? Some people believe that there are many environments in which AGIs can be reached via favourable paths in the loss landscape and dramatically outperform non-AGIs; let’s call this the easy paths hypothesis. By contrast, the hard paths hypothesis is that it’s rare for environments (even complex meta-environments consisting of many separate tasks) to straightforwardly incentivise the development of general intelligence. This would suggest that specific environmental features will be necessary to prevent most models from getting stuck in local minima where they only possess narrow, specialised cognitive skills. There has been a range of speculation on what such features might be - perhaps multi-agent autocurricula, or realistic simulations, or specific types of human feedback. I’ll discuss some of these possibilities later in the post. This spectrum is complicated by its dependence on the model class, training algorithm, and choice of optimiser. If we had a perfect optimiser, then the hilliness of the loss landscape wouldn’t matter. For now, I'm imagining using optimisers fairly similar to current stochastic gradient descent. Meanwhile, I’m assuming in this post that (in accordance with Rich Sutton’s bitter lesson) our models and training algorithms won’t contain very strong inductive biases. In other words: we’ll develop powerful function approximators, but which functions they approximate will primarily be determined by their training environments (and possibly also regularisation, as I’ll discuss later). Arguments for the hard paths hypothesis When predicting AGI timelines, a lot of people focus on progress in compute and algorithms. But I think that environments are more important than they may at first seem, because we have reason to take the hard paths hypothesis seriously. The history of AI is full of realisations that solving high-level tasks is easier than we expect, because those tasks don’t require as much general intelligence as we thought (as highlighted by Moravec’s paradox). Chess doesn’t, Go doesn’t, Starcraft doesn’t. Rather, when we train on these sorts of environments, we get agents with narrow intelligence that is only useful in that environment. The lesson here is that neural networks are very good at doing exactly what we give them feedback to do - even when that feedback is random, large neural networks are capable of simply memorising a lot of information! Another way of phrasing this point: each time we evaluate the training loss, that’s based on a model’s performance on a specific task. So we don’t have any principled way of rewarding models for doing so in a way that generalises to a wide range of unseen tasks. This is in theory a similar problem to making models generalise from the training set to the test set, but in practice much broader - since for an AI to be generally intelligent, it will need to be able to generalise to tasks that are very diffe...