Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Tech company singularities", and steering them to reduce x-risk, published by Andrew Critch on May 13, 2022 on The Effective Altruism Forum. The purpose of this post (also available on LessWrong) is to share an alternative notion of “singularity” that I’ve found useful in timelining/forecasting. A fully general tech company is a technology company with the ability to become a world-leader in essentially any industry sector, given the choice to do so — in the form of agreement among its Board and CEO — with around one year of effort following the choice. Notice here that I’m focusing on a company’s ability to do anything another company can do, rather than an AI system's ability to do anything a human can do. Here, I’m also focusing on what the company can do if it so chooses (i.e., if its Board and CEO so choose) rather than what it actually ends up choosing to do. If a company has these capabilities and chooses not to use them — for example, to avoid heavy regulatory scrutiny or risks to public health and safety — it still qualifies as a fully general tech company. This notion can be contrasted with the following: Artificial general intelligence (AGI) refers to cognitive capabilities fully generalizing those of humans. An autonomous AGI (AAGI) is an autonomous artificial agent with the ability to do essentially anything a human can do, given the choice to do so — in the form of an autonomously/internally determined directive — and an amount of time less than or equal to that needed by a human. Now, consider the following two types of phase changes in tech progress: A tech company singularity is a transition of a technology company into a fully general tech company. This could be enabled by safe AGI (almost certainly not AAGI, which is unsafe), or it could be prevented by unsafe AGI destroying the company or the world. An AI singularity is a transition from having merely narrow AI technology to having AGI technology. I think the tech company singularity concept, or some variant of it, is important for societal planning, and I’ve written predictions about it before, here: 2021-07-21 — prediction that a tech company singularity will occur between 2030 and 2035; 2022-04-11 — updated prediction that a tech company singularity will occur between 2027 and 2033. A tech company singularity as a point of coordination and leverage The reason I like this concept is that it gives an important point of coordination and leverage that is not AGI, but which interacts in important ways with AGI. Observe that a tech company singularity could arrive before AGI, and could play a role in preventing AAGI, e.g., through supporting and enabling regulation; enabling AGI but not AAGI, such as if tech companies remain focussed on providing useful/controllable products (e.g., PaLM, DALL-E); enabling AAGI, such as if tech companies allow experiments training agents to fight and outthink each other to survive. after AGI, such as if the tech company develops safe AGI, but not AAGI (which is hard to control, doesn't enable the tech company to do stuff, and might just destroy it). Points (1.1) and (1.2) are, I think, humanity’s best chance for survival. Moreover, I think there is some chance that the first tech company singularity could come before the first AI singularity, if tech companies remain sufficiently oriented on building systems that are intended to be useful/usable, rather than systems intended to be flashy/scary. How to steer tech company singularities? The above suggests an intervention point for reducing existential risk: convincing a mix of scientists regulators investors, and the public . to shame tech companies for building useless/flashy systems (e.g., autonomous agents trained in evolution-like environments to exhibit survival-oriented intelligence), so they remain focussed on b...