Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why agents are powerful, published by Daniel Kokotajlo on June 6, 2022 on The AI Alignment Forum. [Written for Blog Post Day. Not super happy with it, it’s too rambly and long, but I’m glad it exists.] Here are some questions I think this theory of agency can answer: What are agents? Why should we expect AI agents to be useful for various important tasks? Why should we think agentic mesa-optimizers may arise for some data+reward-signals that weren’t designed explicitly to produce them? Why should we think humans+AI tools will eventually be outcompeted by AI agents? Why should we expect, on priors, mildly superhuman AI agents to be powerful enough to take over the world, if they wanted to? What are agents? Earlier in “Agents as P2B chain reactions” I defined agents as successful P2B feedback loops—a learner algorithm and a planner algorithm hooked up together, the planner plans to plan better and thus outputs actions that result in getting more useful data into the learner, more resources for the planner to plan with. then the process repeats. Like how “fire” is a successful feedback loop in which heat+plasma+fuel come together to produce more heat+plasma, and envelop more fuel, if any is nearby. Then I asked: Is the planner algorithm strictly necessary here? Could you get something similar to a P2B feedback loop, but built around something other than a planner? I explored this question indirectly in “Gradations of Agency.” The answer seems to be “sorta.” You can have e.g. a Level 3 system that doesn’t do any planning and yet behaves remarkably similar to a P2B feedback loop in many contexts, due to imitating the success of others. Such a system would tend to underperform P2B in contexts where there aren’t good examples to imitate.We could stick to the original definition and say: Agents are P2B feedback loops, so, level 4 and above. But I think it’s probably better to be a bit more galaxy brain and say: the key thing is convergent instrumental resource feedback loops and that thinking about P2B is a stepping stone which helps us see why such phenomena exist and why the really big ones tend to have planners sitting in the middle directing them, and the really really big ones tend to have level 6+ algorithms sitting in the middle directing them. By analogy, heat+plasma+wood+oxygenl chain reactions aren’t the key thing. The key thing is chain reactions that convert some sort of fuel into thermal and kinetic energy plus more of the chain reaction. Heat+plasma+wood+oxygen is a particularly important one, due to the ease of creating it and the abundance of fuel for it on Earth, but there’s also e.g. baking soda + vinegar and neutrons+uranium. And also self-replicating nanobots + pretty much anything. Similarly: The important thing for understanding agency is understanding that the world contains various self-sustaining chain reactions that feed off instrumental resources like data, money, political power, etc. and spread to acquire more such resources. Level 4 P2B loops are a particularly important, powerful, and widespread instance of this type, analogous to fire, but there are less powerful/important things (such as Level 3, Level 2.) and also more powerful/important things (Level 6 and above?). Why should we expect AI agents to be useful for various important tasks? Why should we think agentic mesa-optimizers may arise for some data+reward-signals that weren’t designed explicitly to produce them? Both questions have the same answer. Agency works; it’s powerful and general. Recalling this graph and the conjectures from last post, I claim that as you crank up the “how fast does this thing learn, how high a score does it get, in how large and diverse a range of environments?” dial, you ascend the hierarchy until you get agents. You can get powerful non-agents but only by d...