Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: autonomy: the missing AGI ingredient?, published by nostalgebraist on May 25, 2022 on The AI Alignment Forum. Epistemic status: trying to feel out the shape of a concept and give it an appropriate name. Trying to make explicit some things that I think exist implicitly in many people's minds. This post makes truth claims, but its main goal is to not to convince you that they are true. Here are some things I would expect any AGI to be able to do: Operate over long intervals of time relative to its sensory bandwidth (e.g. months or years of ~30 fps visual input). Remember specific sensory experiences from long ago that are relevant to what's happening to it now. (E.g. remember things it saw months or years ago.) Retain or forget information and skills over long time scales, in a way that serves its goals. E.g. if it does forget some things, these should be things that are unusually unlikely to come in handy later. Re-evaluate experiences that happened a long time ago (e.g. years ago) in light of newer evidence (observed in e.g. the last hour), and update its beliefs appropriately. Continually adjust its world model in light of new information during operation. E.g. upon learning that a particular war has ended, it should act as though the war is not happening, and do so in all contexts/modalities. As with humans, this adaptation may take a nonzero amount of time, during which it might "forget" the new fact sometimes. However, adaptation should be rapid enough that it does not impede acting prudently on the most relevant implications of the new information. This may require regular "downtime" to run offline training/finetuning (humans have to sleep, after all). But if so, it should require less than 1 second of downtime per second of uptime, ideally much less. Perform adjustments to itself of the kind described above in a "stable" manner, with a negligibly low rate of large regressions in its knowledge or capabilities. E.g. if it is updating itself by gradient descent, it should do so in a way that avoids (or renders harmless) the gradient spikes and other instabilities that cause frequent quality regression in the middle of training for existing models, especially large ones. Keep track of the broader world context while performing a given task. E.g. an AGI playing a video game should not forget about its situation and goals in the world outside the game. It might "get distracted" by the game (as humans do), but it should have some mechanism for stopping the game and switching to another task if/when its larger goals dictate that it should do so, at least some of the time. Maintain stable high-level goals across contexts. E.g. if it is moved from one room to another, very different-looking room, it should not infer that it is now "doing a different task" and ignore all its previously held goals. I'm not sure how related these properties are, though they feel like a cluster in my mind. In any case, a unifying theme of this list is that current ML models generally do not do these things -- and we do not ask them to do these things. We don't train models in a way that encourages these properties, and in some cases we design models whose structures rule them out. Benchmarks for these properties are either nonexistent, or much less mature than more familiar benchmarks. Is there an existing name for this cluster? If there isn't one, I propose the name "autonomy." This may not be an ideal name, but it's what I came up with. I think this topic is worthy of more explicit discussion than it receives. In debates about the capabilities of modern ML, I usually see autonomy brought up in a tangential way, if at all. ML detractors sometimes cite the lack of autonomy in current models as a flaw, but they rarely talk about the fact that ML models are not directly trained to do any of th...